Category Archives: Akka

Last time we looked at Akka Http, this time we will look at Akka Streams.

Akka Streams is a vast topic, and you will definitely need to supplement this post with the official documentation.

Akka Streams is one of the founding members of Reactive Streams, and Akka streams is one implementation (there are many) of the Reactive Streams APIs.

Reactive Streamsis an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments (JVM and JavaScript) as well as network protocols.

Introduction

There may be some readers who have come from .NET such as myself who have used RX.

You may even have heard of Reactive Streams before. So what exactly makes reactive streams different from Rx?

The central thing that is the big win with reactive streams over Rx is the idea of back pressure. Here is what the Akka docs say about back pressure

The back pressure protocol is defined in terms of the number of elements a downstream Subscriber is able to receive and buffer, referred to as demand. The source of data, referred to as Publisher in Reactive Streams terminology and implemented as Source in Akka Streams, guarantees that it will never emit more elements than the received total demand for any given Subscriber.

Luckily this is all inbuilt to Akka streams, you do not have to worry about this too much as a user of Akka streams.

You can pretty much decide how you want the built in streams pipelines (which we will be diving into in more details below) in terms of backpressure using the OverflowStrategy enum value. Here is a very simple example

Source(1 to 10).buffer(10, OverflowStrategy.backpressure)

Where the following are the available OverflowStrategy values

object OverflowStrategy {
/**
* If the buffer is full when a new element arrives, drops the oldest element from the buffer to make space for
* the new element.
*/
def dropHead: OverflowStrategy = DropHead
/**
* If the buffer is full when a new element arrives, drops the youngest element from the buffer to make space for
* the new element.
*/
def dropTail: OverflowStrategy = DropTail
/**
* If the buffer is full when a new element arrives, drops all the buffered elements to make space for the new element.
*/
def dropBuffer: OverflowStrategy = DropBuffer
/**
* If the buffer is full when a new element arrives, drops the new element.
*/
def dropNew: OverflowStrategy = DropNew
/**
* If the buffer is full when a new element is available this strategy backpressures the upstream publisher until
* space becomes available in the buffer.
*/
def backpressure: OverflowStrategy = Backpressure
/**
* If the buffer is full when a new element is available this strategy completes the stream with failure.
*/
def fail: OverflowStrategy = Fail
}

So that is the basic idea, Akka streams does provide a lot of stuff, such as

Built in stages/shapes

A graph API

Ability to create your own stages/shapes

For the rest of this post we will be looking at some examples of these 3 points.

Working With The Akka Streams APIs

As stated at the beginning of this post the Akka Streams implementation is vast. There is a lot of ground to cover, far more than I can reasonably cover in a small blog post. The official docs are still the place to go, but if you have not heard of Akka Streams this post may be enough to get you into it.

Working With Built In Stages/Shapes

Akka comes with loads of prebuilt stages which we can make use of. However before I mention those lets try and just spend a bit of time taking a bit about how you use the Akka Streams APIs in their most basic form.

The idea is that we have 4 different parts that make up a useable pipeline.

Working With The Graph API

Akka streams also comes with a pretty funky graph building DSL. You would use this when you want to create quite elaborate flows.

The other very interesting thing about the graph builder DSL is that you can use custom shapes inside it, and you can also leave it partially connected. Such that you could potentially use it as a Source/Sink.

Lets say you had an output from the graph you built using the graph DSL, you could then use that partially constructed graph as a Source in its own right.

The same goes if you had an unconnected input in the graph you created you could use that as a Sink.

Most of the actual logic will be inside the createLogic method. But in order to do anything useful in there you will need to use handlers. Handlers are what you use to handle input/output. There are InHandler and OutHandler.

Each of which has its own state machine flow. For example this is the state machine for an OutHandler

Where Can I Find The Code Examples?

Last time we talked about routing within Akka. This time we will be looking at Akka’s support for http.

But just before that, a bit of history. Before Akka.Http there was already a fairly successful Akk based http option available to you as a Scala developer, called Spray. There is a lot of Spray documentation available here http://spray.io/

This framework was extremely well thought of, so much so that the good people at Akka have taken on much of the good work done by this team, and it now forms much of the codebase for Akka Http.

In fact if you are familiar with Spray, you will certainly notice quite a lot of similarities in the way routes and JSON are handled in Akka.Http, as it is pretty much the Spray code.

Introduction

Akka.Http comes with server side and client side libraries. It also comes with good support for standard serialization such as JSON/XML and the ability to roll your own serialization should you want to.

It also comes with a fairly nifty routing DSL which is very much inspired by the work done in Spray.

This post will concentrate on the common use cases that you may come across when working with HTTP.

SBT Dependencies

As usual we need to make sure we have the correct JARs referenced. So here is the SBT file that I am using for both the server side/client side and common messages that pass between them

It can be seen that there are some common routing DSL bits and bobs in there, such as:

path : which satisfies the route name part of the route

get : which tells us that we should go further into the route matching if it’s a GET http request and it matched the path route DSL part

post: which tells us that we should go further into the route matching if it’s a POST http request and it matched the path route DSL part

complete : This is the final result from the route

These parts of the DSL are known as directives. The general anatomy of a directive is as follows:

name(arguments) { extractions =>
... // inner route
}

It has a name, zero or more arguments and optionally an inner route (The RouteDirectives are special in that they are always used at the leaf-level and as such cannot have inner routes). Additionally directives can “extract” a number of values and make them available to their inner routes as function arguments. When seen “from the outside” a directive with its inner route form an expression of type Route.

So again we use the path/get directives, but this time we complete with an Item. This is done due to the JSON support that is able to create the right serialization data for us. We will look at this in the next section

So again we use the path directive, but this time we use a post, where the post expects an item as JSON to be provided. The converting from the incoming JSON string to an Item is done using an Unmarshaller, we will look at this in the next section

So let’s see this one in postman

JSON Support

Akka.http provides JSON support using this library akka-http-spray-json-experimental which you can grab from Maven Central Repo.

JsonProtocol

When using spray we may use the SprayJsonProtocol and DefaultJsonProtocol to create the JSON protcol for your custom objects

Lets consider the Item class we have seen in the demos so far

package common
final case class Item(name: String, id: Long)

This is how we might write the JSON protocol code for this simple class

Luckily you don’t really have to get that involved with these that often as the routing DSL does most of the heavy lifting for you when you do the complete this is taken care of for you providing there is a marshaller that can be found implicitly

Unmarshalling

Unmarshalling is the process of taking the on the wire format (JSON string in these examples) back into a scala class (Item class in this case)

Luckily you don’t really have to get that involved with these that often as the routing DSL does most of the heavy lifting for you, which is what we use this part of the routing DSL, where this will use an unmarshaller to create the Item from the JSON string on the wire

entity(as[Item]) { item =>

WebSockets

Akka Http also supports web sockets too. Lets start this investigation with looking at what is required from the routing DSL perspective, which starts like this

So we need to supply a flow. A Flow is part of akka reactive streams which will look at in the next part. But for now just be aware that you can create a Flow from a Sink/Source and Materializer to materialize the flow.

Where Can I Find The Code Examples?

Last time we looked at Akka Clustering, this time we will look at routing.

Routing allows messages to be routed to one or more actors known as routees, by sending the messages to a router that will know how to route the messages to the routees.

Akka comes with quite a few inbuilt routing strategies that we can make use of. We will look at these next.

Types Of Routing Strategy

Akka comes with a whole bunch of inbuilt routing strategies such as :

RoundRobin : Routes in a round-robin fashion to its routees.

Random : This router type selects one of its routees randomly for each message.

SmallestMailBox : A Router that tries to send to the non-suspended child routee with fewest messages in mailbox. The selection is done in this order: pick any idle routee (not processing message) with empty mailbox pick any routee with empty mailbox pick routee with fewest pending messages in mailbox pick any remote routee, remote actors are consider lowest priority, since their mailbox size is unknown

Broadcast : A broadcast router forwards the message it receives to all its routees.

ScatterGatherFirstCompleted : The ScatterGatherFirstCompletedRouter will send the message on to all its routees. It then waits for first reply it gets back. This result will be sent back to original sender. Other replies are discarded.

TailChopping : The TailChoppingRouter will first send the message to one, randomly picked, routee and then after a small delay to a second routee (picked randomly from the remaining routees) and so on. It waits for first reply it gets back and forwards it back to original sender. Other replies are discarded.

The goal of this router is to decrease latency by performing redundant queries to multiple routees, assuming that one of the other actors may still be faster to respond than the initial one.

Regular Actor As A Router

Akka allows you to create routers in 2 ways, the first way is to use RoutingLogic to setup your router.

Therere are quite a few specializations of the RoutingLogic, such as

RoundRobinRoutingLogic

RandomRoutingLogic

SmallestMailboxRoutingLogic

BroadcastRoutingLogic

You would typically use this in a regular actor. The actor in which you use the RoutingLogic would be the router. If you go down this path you would be responsible for managing the routers children, ie the routees. That means you would be responsible for managing ALL aspects of the routees, including adding them to a list of available routees, watching them for Termination to remove them from the list of available routees (which sounds a lot like supervision doesn’t it).

Here is what a skeleton for an actor that is setup manually as a router may look like

It can be seen that I pass in the RoutingLogic, which would be one of the available RoutingLogic strategies that akka comes with.

The other thing to note is that as we stated earlier we need to FULLY manage the collection of routee actors ourselves, including watching them for Termination.

Sure there is a better way?

Well yes thankfully there is, Akka also provides a Pool for this job. We will look at that next.

Pool

Akka comes with the ability to create a router using a pool where we tell it what actors we want to use as the routees, how many routees we want, and how the supervision should be handled.

Here is some code from by demo code that uses 2 utility methods to create a pool created router that will use a simple FibboniciActor which is sent messages via an actor that is created using the pool router value

Supervision Using Pool

Routees that are created by a pool router will be created as the router’s children. The router is therefore also the children’s supervisor.

The supervision strategy of the router actor can be configured with the supervisorStrategy property of the Pool. If no configuration is provided, routers default to a strategy of “always escalate”. This means that errors are passed up to the router’s supervisor for handling. The router’s supervisor will decide what to do about any errors.

Note the router’s supervisor will treat the error as an error with the router itself. Therefore a directive to stop or restart will cause the router itself to stop or restart. The router, in turn, will cause its children to stop and restart.

It should be mentioned that the router’s restart behavior has been overridden so that a restart, while still re-creating the children, will still preserve the same number of actors in the pool.

This means that if you have not specified supervisorStrategy of the router or its parent a failure in a routee will escalate to the parent of the router, which will by default restart the router, which will restart all routees (it uses Escalate and does not stop routees during restart). The reason is to make the default behave such that adding withRouter to a child’s definition does not change the supervision strategy applied to the child. This might be an inefficiency that you can avoid by specifying the strategy when defining the router.

Group

You may also wish to create your routees separately and let the router know about them. This is achievable using Groups. This is not something I decided to cover in this post, but if this sounds of interest to you, you can read more about it at the official documentation here:

We get this output, where the routee with the smallest mailbox will get the message sent to it. This example may look a bit weird, but if you think about it, by the time the new message is sent the 1st routee (workerActor0) will have dealt with the 1st message, and it ready to receive a new one, and since it’s the 1st routee in the list it is still considered the one with the smallest mailbox. If you introduced an artificial delay in the actor dealing with the message it may show different more interesting results.

So that about covers the demos I have created for using your own actor and using the RoutingLogic. Lets now look at using pools, as I have stated already pools take care of supervision for us, so we don’t have to manually take care of that any more.

As before I have a helper actor to work with the pool, that accepts the router, where the router will receive the messages to send to its routees.

Here is the output when we run this. It can be seen that we simply get the results from the routee that completed first

FibonacciActor : ($d) -> has been asked to calculate FibonacciNumberFibonacciActor : ($e) -> has been asked to calculate FibonacciNumberFibonacciActor : ($a) -> has been asked to calculate FibonacciNumberFibonacciActor : ($c) -> has been asked to calculate FibonacciNumberFibonacciActor : ($b) -> has been asked to calculate FibonacciNumberFibonacciActor : ($d) came back with result -> 55

TailChoppingPool

Here is the output when we run this. It can be seen that we simply get the results from the routee that completed first, out of the few routees that the message was sent to

FibonacciActor : ($b) -> has been asked to calculate FibonacciNumberFibonacciActor : ($b) came back with result -> 55

What About Custom Routing Strategy

Akka allows you to create your own routing strategy where you would create a class that extends the inbuilt Akka RoutingLogic. You can read more about this in the official Akka documentation:

Where Can I Find The Code Examples?

Last time we look at remoting. You can kind of think of clustering as an extension to remoting, as some of the same underlying parts are used. But as we will see clustering is way more powerful (and more fault tolerant too).

My hope is by the end of this post that you will know enough about Akka clustering that you would be able to create your own clustered Akka apps.

A Note About All The Demos In This Topic

I wanted the demos in this section to be as close to real life as possible. The official akka examples tend to have a single process. Which I personally think is quite confusing when you are trying to deal with quite hard concepts. As such I decided to go with multi process projects to demonstrate things. I do however only have 1 laptop, so they are hosted on the same node, but they are separate processes/JVMs.

I am hoping by doing this it will make the learning process easier, as it is closer to what you would do in real life rather than have 1 main method that spawns an entire cluster. You just would not have that in real life.

What Is Akka Clustering?

Unlike remoting which is peer to peer, a cluster may constitute many members, which can grow and contract depending on demand/failure. There is also the concept of roles for actors with a cluster, which this post will talk about.

You can see how this could be very useful, in fact you could see how this may be used to create a general purpose grid calculation engine such as Apache Spark.

Seed Nodes

Akka has the concept of some initial contact points within the cluster to allow the cluster to bootstrap itself as it were.

Here is what the official Akka docs say on this:

You may decide if joining to the cluster should be done manually or automatically to configured initial contact points, so-called seed nodes. When a new node is started it sends a message to all seed nodes and then sends join command to the one that answers first. If no one of the seed nodes replied (might not be started yet) it retries this procedure until successful or shutdown.

You may choose to configure these “seed nodes” in code, but the easiest way is via configuration. The relevant part of the demo apps configuration is here

The seed nodes can be started in any order and it is not necessary to have all seed nodes running, but the node configured as the first element in the seed-nodes configuration list must be started when initially starting a cluster, otherwise the other seed-nodes will not become initialized and no other node can join the cluster. The reason for the special first seed node is to avoid forming separated islands when starting from an empty cluster. It is quickest to start all configured seed nodes at the same time (order doesn’t matter), otherwise it can take up to the configured seed-node-timeout until the nodes can join.

Once more than two seed nodes have been started it is no problem to shut down the first seed node. If the first seed node is restarted, it will first try to join the other seed nodes in the existing cluster.

We will see the entire configuration for the demo app later on this post. For now just be aware that there is a concept of seed nodes and the best way to configure those for the cluster is via configuration.

Saying that there may be some amongst you that would prefer to use the JVM property system which you may do as follows:

Roles

Akka clustering comes with the concept of roles.You may be asking why would we need that?

Well its quite simple really, say we have a higher than normal volume of data coming through you akka cluster system, you may want to increase the total processing power of the cluster to deal with this. How do we do that, we spin up more actors within a particular role. The role here may be “backend” that do work designated to them by some other actor say “frontend” role.

By using roles we can manage which bits of the cluster get dynamically allocated more/less actors.

You can configure the minimum number of role actor in configuration, which you can read more about here:

We will see more on this within the demo code which we will walk through later

ClusterClient

What use is a cluster which cant receive commands from the outside world?

Well luckily we don’t have to care about that as Akka comes with 2 things that make this otherwise glib situation ok.

Akka comes with a ClusterClient which allows actors which are not part of the cluster to talk to the cluster. Here is what the offical Akka docs have to say about this

An actor system that is not part of the cluster can communicate with actors somewhere in the cluster via this ClusterClient. The client can of course be part of another cluster. It only needs to know the location of one (or more) nodes to use as initial contact points. It will establish a connection to a ClusterReceptionist somewhere in the cluster. It will monitor the connection to the receptionist and establish a new connection if the link goes down. When looking for a new receptionist it uses fresh contact points retrieved from previous establishment, or periodically refreshed contacts, i.e. not necessarily the initial contact points.

Receptionist

As mentioned above the ClusterClient makes use of a ClusterReceptionist, but what is that, and how do we make a cluster actor available to the client using that?

The ClusterReceptionist is an Akka contrib extension, and must be configured on ALL the nodes that the ClusterClient will need to talk to.

There are 2 parts this, firstly we must ensure that the ClusterReceptionist is started on the nodes that ClusterClient will need to communicate with. This is easily done using the following config:

The other thing that needs doing, is that any actor within the cluster that you want to be able to talk to using the ClusterClient will need to register itself as a service with the ClusterClientReceptionist. Here is an example of how to do that

The “Official” example as it is, provides a cluster which contains “frontend” and “backend” roles. The “frontend” actors will take a text message and pass it to the register workers (“Backend”s) who will UPPERCASE the message and return to the “frontend”.

I have taken this sample and added the ability to use the ClusterClient with it, which works using Future[T] and the ask pattern, such that the ClusterClient will get a response from the cluster request.

We will dive into all of this in just a moment

For the demo this is what we are trying to build

SBT / Dependencies

Before we dive into the demo code (which as I say is based largely on the official lightbend clustering example anyway) I would just like to dive into the SBT file that drives the demo projects

frontend : frontend cluster based actors (the client will talk to these)

backend : backend cluster based actors

The Projects

Now that we have seen the projects involved from an SBT point of view, lets continue to look at how the actual projects perform their duties

Remember the workflow we are trying to achieve is something like this

We should ensure that a frontend (seed node) is started first

We should ensure a backend (seed node) is started. This will have the effect of the backend actor registering itself as a worker with the already running frontend actor

At this point we could start more frontend/backend non seed nodes actors, if we chose to

We start the client app (root) which will periodically send messages to the frontend actor that is looked up by its known seed node information. We would expect the frontend actor to delegate work of to one of its known backend actors, and then send the response back to the client (ClusterClient) where we can use the response to send to a local actor, or consume the response directly

Common

The common project simply contains the common objects across the other projects. Which for this demo app are just the messages as shown below

As you can see this is a regular actor, but there are several important things to note here:

We setup the ClusterClient with a known set of seed nodes that we can expect to be able to contact within the cluster (remember these nodes MUST have registered themselves as available services with the ClusterClientReceptionist

That we use a new type of actor a ClusterClient

That we use the ClusterClient to send a message to a seed node within the cluster (frontend) in our case. We use the ask pattern which will give use a Future[T] which represents the response.

We use the response to send a local message to ourself

FrontEnd

As previously stated the “frontend” role actors serve as the seed nodes for the ClusterClient. There is only one seed node for the frontend which we just saw the client app uses via the ClusterClient.

So what happens when the client app uses the frontend actors via the ClusterClient, well its quite simple the client app (once a connection is made to the frontend seed node) send a simple TransformationJob which is a simple message that contains a bit of text that the frontend actor will pass on to one of its registered backend workers for processing.

The backend actor (also in the cluster) will simply convert the TransformationJob contained text to UPPERCASE and return it to the frontend actor. The frontend actor will then send this TransformationResult back to the sender which happens to be the ClusterClient. The client app will listen to this (which was done using the ask pattern) and will hook up a callback for the Future[T] and will the send the TransformationResult to the clients own actor.

Happy days.

So that is what we are trying to achieve, lets see what bits and bobs we need for the frontend side of things

The important parts here are that we embellish the read config with the role of “frontend”, and that we also register the frontend actor with the ClusterClientReceptionist such that the actor is available to communicate with by the ClusterClient

Other than that it is all pretty vanilla akka to be honest

So lets now focus our attention to the actual frontend actor, which is shown below

That when a backend registers it will send a BackendRegistration, which we then watch and monitor, and if that backend terminates it is removed from the list of this frontend actors known backend actors

That we palm off the incoming TransformationJob to a random backend, and then use the pipe pattern to pipe the response back to the client

And with that, all that is left to do is examine the backend code, lets looks at that now

BackEnd

As always lets start with the configuration, which for the backend is as follows:

That we use the cluster events, to subscribe to MemberUp such that if its “frontend” role actor, we will register this backend with it by sending a BackendRegistration message to it

That for any TrasformationJob received (from the frontend which is ultimately for the client app) we do the work, and send a TransformationResult back, which will make its way all the way back to the client

And in a nutshell that is how the entire demo hangs together. I hope I have not lost anyone along the way.

Anyway lets now see how we can run the demo

How do I Run The Demo

You will need to ensure that you run the following 3 projects in this order (as a minimum. You can run more NON seed node frontend/backend versions before you start the root (client) if you like)

Frontend (seed node) : frontend with command line args : 2551

Backend (seed node) : backend with command line args : 2551

Optionally run more frontend/backend projects but DON’T supply any command line args. This is how you get them to not be treated as seed nodes

Where Can I Find The Code Examples?

It has been a while since I wrote a post, the reason for this is actually this post.

I would consider remoting/clustering to be some of the more advanced stuff you could do with Akka. That said this and the next post will outline all of this good stuff for you, and by the end of this post I would hope to have demonstrated enough for you guys to go off and write Akka remoting/clustered apps.

I have decided to split the remoting and clustering stuff into 2 posts, to make it more focused and digestible. I think this is the right thing to do.

A Note About All The Demos In This Topic

I wanted the demos in this section to be as close to real life as possible. The official akka examples tend to have a single process. Which I personally think is quite confusing when you are trying to deal with quite hard concepts. As such I decided to go with multi process projects to demonstrate things. I do however only have 1 laptop, so they are hosted on the same node, but they are separate processes/JVMs.

I am hoping by doing this it will make the learning process easier, as it is closer to what you would do in real life rather than have 1 main method that spawns an entire cluster. You just would not have that in real life.

What Is Akka Remoting

If you have ever used RMI in Java or Remoting/WCF in C# you can kind of think of Akka remoting as something similar to that. Where there is the ability to call a remote objects method as is it were local. It is essentially peer-to-peer.

Obviously in Akkas case the remote object is actually an Actor, and you will not actually be calling a method at all,but will instead by treating the remote actor just like any other actor where you simply pass messages to it, and the remote actor will work just like any other actor where it will receive the message and act on it accordingly.

This is actually quite unique actually, I have work with Java Remoting and also C# Remoting, and done a lot with .NET WCF. What all of these had in common was that there was some code voodoo that you had to do, where the difference between working with a local object and working with a remote object required a fair bit of code, be it remoting channels, proxies etc etc

In Akka there is literally no change in coding style to work with remoting, it is completely configuration driven. This is quite nice.

Akkas Remoting Interaction Models

Akka supports 2 ways of using remoting

Lookup : Where we use actorSelection to lookup an already running remote actor

Creation : Where an actor will be created on the remote node

We will be looking at both these approaches

Requirements

As I have stated on numerous occasions I have chosen to use SBT. As such this is my SBT file dependencies section for both these Remoting examples.

And that is all there is to the remote side of the “remote selection” remoting version. Lets now turn our attention to the local side.

Local Project

So far we have a remote actor which is configured to up and running at 127.0.0.1:4444.

We now need to open up the local side of things. This is done using the following configuration.Notice the port is a different port from the already in use 4444. Obviously if you host these actors on physically different boxes there would be nothing to stop you using port 4444 again, but from a sanity point of view, I find it is better to not do that.

Plain Selection

We can make use of plain old actor selection to select any actor by a path of our chosing. Where we may use the resolveOne (if we expect to only match one actor, remember we can use wildcards so there are times we may match more than one) to give us a ActorRef.

context.actorSelection(path).resolveOne()

When we use resolveOne() we would get a Future[ActorRef] that we can use in any of the normal ways we would handle and work with Futute[T]. I have chosen to use a for comprehension to capture the result of the ActorRef of the more actor. I also monitor the remote actor using context.watch such that if it terminates we will see a Terminated message and can shutdown the local actor system.

We also make use of the become (see the state machines post for more info on that) to swap out the message loop for the local actor, so work differently once we have a remote ActorRef.

Once we have an ActorRef representing the remote actor it is pretty standard stuff where we just send messages to the remote actor ref using the ActorRef that represents it.

Here is the entire code for the plain actor selection approach of dealing with a remote actor.

Using Identity Messages

Another approach that can be taken rather than relying on straight actor selection is by using some special Akka messages, namely Identify and ActorIdentity.

The idea is that we still use actorSelection for a given path, but rather than using resolveOne we sent the send the ActorSelection a special Identify message. The actor that was chosen by the ActorSelection should see this Identify message and should respond with a ActorIdentity message.

As this point the local actor can simply listen for ActorIdentity messages and when it sees one, it can test this messages correlationId to see if it matches the requested path, if it does you know that is the correct actor and you can then use the ActorRef of the ActorIdentity message.

As in the previous example we also make use of the become (see the state machines post for more info on that) to swap out the message loop for the local actor, so work differently once we have a remote ActorRef.

Once we have an ActorRef representing the remote actor it is pretty standard stuff where we just send messages to the remote actor ref using the ActorRef that represents it.

How do I Run The Demo

You will need to ensure that you run the following 2 projects in this order:

Remote

Local

Once you run the 2 projects you should see some output like this

The Remote project output:

[INFO] [10/03/2016 07:02:54.282] [main] [akka.remote.Remoting] Starting remoting[INFO] [10/03/2016 07:02:54.842] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://RemoteDemoSystem@127.0.0.1:4444][INFO] [10/03/2016 07:02:54.844] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://RemoteDemoSystem@127.0.0.1:4444]RemoteActor received message ‘The RemoteActor is alive’[INFO] [10/03/2016 07:02:54.867] [RemoteDemoSystem-akka.actor.default-dispatcher-15] [akka://RemoteDemoSystem/deadLetters] Message [java.lang.String] from Actor[akka://RemoteDemoSystem/user/RemoteActor#-109465353] to Actor[akka://RemoteDemoSystem/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.RemoteActor received message ‘Hello from the LocalActorUsingPlainSelection’RemoteActor received message ‘Hello back to you’RemoteActor received message ‘Hello back to you’RemoteActor received message ‘Hello back to you’RemoteActor received message ‘Hello back to you’RemoteActor received message ‘Hello back to you’

The Local project output:

[INFO] [10/03/2016 07:03:09.489] [main] [akka.remote.Remoting] Starting remoting[INFO] [10/03/2016 07:03:09.961] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://LocalDemoSystem@127.0.0.1:64945][INFO] [10/03/2016 07:03:09.963] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://LocalDemoSystem@127.0.0.1:64945]Resolved remote actor ref using SelectionLocalActorUsingPlainSelection received message: ‘Hello from the RemoteActor’LocalActorUsingPlainSelection received message: ‘Hello from the RemoteActor’LocalActorUsingPlainSelection received message: ‘Hello from the RemoteActor’LocalActorUsingPlainSelection received message: ‘Hello from the RemoteActor’LocalActorUsingPlainSelection received message: ‘Hello from the RemoteActor’LocalActorUsingPlainSelection received message: ‘Hello from the RemoteActor’

Remote Creation

Just as we did with remote selection, remote creation shall be split into a remote project and a local project, however since this time the local project must know about the type of the more actor to create it in the first place we introduce a common project which both the remote and local depend on.

In this example there will be 3 projects configured in the SBT file

Remote : The remote actor, that is expected to be created by the local actor

I also mentioned that the remote actor system MUST be started to allow remote creation to work, as such the entire codebase for the remote end of thing (excluding the actor remote actor which gets created by the local side of thing) is shown below

All that is happening here is that the remote actor system gets created.

Local Project

Most of the hard work is done in this project. As it is the local side of things that is responsible for creating and deploying the remote actor in the remote actor system, before it can then make use of it.

Lets start with the deployment of the remote actor from the local side.

If you look carefully at this configuration file and the one in the remote end you will see that the ip address/poprt/actor system name used within the deployment section all match. This is how the local actor system is able to create and deploy an actor to the remote actor system (which must be running prior to the local actor system trying to deploy a remote actor to it)

So now that we have seen this config, lets see how it is used by the local

It can be seen that the first thing we try and do is try and create the remote actor (CreationActor) using the config above. If this all works we will end up with a CreationActor being created in the already running remote actor system. This CreationActor can then be used just like any other actor.

I have chosen to use scalatest, but you could use other popular testing frameworks such as specs2.

Testing Actors

The following sub sections will outline how Akka allows the testing of actors

The built in ‘testActor’

The official Akka testkit docs do a great job of explaining the testActor and why there is a need for a test actor.

Testing the business logic inside Actor classes can be divided into two parts: first, each atomic operation must work in isolation, then sequences of incoming events must be processed correctly, even in the presence of some possible variability in the ordering of events. The former is the primary use case for single-threaded unit testing, while the latter can only be verified in integration tests.

Normally, the ActorRef shields the underlying Actor instance from the outside, the only communications channel is the actor’s mailbox. This restriction is an impediment to unit testing, which led to the inception of the TestActorRef.

It is by using this testActor that we are able to test the individual operations of an actor under test.

You essentially instantiate the TestActorRef passing it the real actor you would like to test. The test actor then allows you to send messages which are forwarded to the contained real actor that you are attempting to test.

CallingThreadDispatcher/TestActorRef

As Akka is an asynchronous beast by nature, and uses the concept of Dispatchers to conduct the dispatching of messages. We have also seen that the message loop (receive) can be replaced with become/unbecome, all of which contributes to the overall behviour of the actor being quite hard to test.

Akka comes with a special actor called TestActorRef.Which is a special actor that comes with the Akka TestKit.

It should come as no surprise that this TestActorRef aslo makes use of a Dispatcher. But what makes this actor more suited to testing is that it uses a specialized testing Dispatcher, which makes testing the asynchronous code easier to test.

The specialized dispatcher is called CallingThreadDispatcher. As the name suggests it uses the current thread to deal with the message dispatching.

This makes thing easier of that there is no doubt. As stated you don’t really need to do anything other than use the Akka TestKit TestActorRef.

Anatomy Of An Actor Testkit Test

This is what a basic skeleton looks like when using the Akka TestKit (please note this is for ScalaTest)

It can be seen that we make use of the TestActorRef (which we discussed above the one that uses the CallingThreadDispatcher), as the actor that we use to wrap (for want of a better word) the actual actor we wish to test.

Expecting Exceptions

Another completely plausible thing to want to do is test for exceptions that may be thrown. It can be seen in the HelloActor that we are trying to test that it will throw an IllegalArgumentException should it see a message it doesn’t handle.

So how do we test that?

We use the inbuilt intercept function to allow us to catch the exception

This example is fairly good as it only has 2 states On/Off. So it makes for quite a good simply example to showcase the testing.

Another Special Test Actor

To test FSMs there is yet another specialized actor which may only be used for testing FSMs. This actor is call TestFSMRef. Just like the TestActorRef you use the TestFSMRef to accept the actual FSM actor you are trying to test.

Here is an example of that

val fsm = TestFSMRef(new LightSwitchActor())

The TestFSMRef comes with a whole host of useful methods, properties that can be used when testing FSMs. We will see some of them used below.

Testing Initial State

As we saw last time Akka FSM has the idea of an initialise() method that may be used to place the FSM in an initial state. So we should be able to test that.

Testing StateTimeout

Another thing that AkkaFSM supports is the notion of a StateTimeout. In the example FSM we are trying to test, if the FSM stays in the On state for more than 1 second it should automatically move to the Off state.

Testing Using Probes

So far we have been looking at testing a single actor that might reply to a single sender. Sometimes though we may need to test an enture suite of actors all working together. And due to the single threaded nature of the TestActorRef (thanks to the very useful CurrentThreadDispatcher), we may find it difficult to distinguish the incoming messages read.

Akka TestKit provides yet another abstraction to deal with this, which is the idea of a concrete actor that you inject into the message flow. This concept is called a TestProbe.

It can be seen that the TestProbe comes with its own set of useful assertion methods. This is due to the fact that TestProbe inherits from the TestKit trait, and as such you can expect to find ALL the TestKit traits assertions available to use when using TestProbe objects.

Where Can I Find The Code Examples?

I will be augmenting this GitHub repo with the example projects as I move through this series

In this post we will look at 2 ways you can write state machines with Akka. We will firstly examine the more primitive (but easily understandable) approach, and then look into the more sophisticated approach offered by AkkaFSM.

What Is A State Machine?

For those of you out there that do not know what a state machine is.

This is what Wikipedia says about them

A finite-state machine (FSM) or finite-state automaton (FSA, plural: automata), or simply a state machine, is a mathematical model of computation used to design both computer programs and sequential logic circuits. It is conceived as an abstract machine that can be in one of a finite number of states. The machine is in only one state at a time; the state it is in at any given time is called the current state. It can change from one state to another when initiated by a triggering event or condition; this is called a transition. A particular FSM is defined by a list of its states, and the triggering condition for each transition.

UnBecome

The other way to swap out the message loop relies on having matching pairs of become/unbecome. Where the standard message loop is not replaced as such, but will use the last value on a stack of values as the message loop.

Care must taken to ensure the amount of push (become) and pop (unbecome) operations match, otherwise memory leaks may occur. Which is why this is not the default behavior.

Here is an example actor that uses the become/unbecome matched operations.

AkkaFSM

While become/unbecome will get the job done. Akka comes with a much better alternative called Akka FSM

Using Akka FSM you can not only handle states but also have state data, and add code against the movement from one state to the next, which as most people will know are known as “transitions”.

When using Akka FSM you may see a state machine expressed using this sort of notation

State(S) x Event(E) -> Actions (A), State(S’)

f we are in state S and the event E occurs, we should perform the actions A and make a transition to the state S’.

To use Akka FSM there are a number of things you can do. Some of them are mandatory and some you can opt into depending on your requirements. Lets have a look at some of the moving parts that make Akka FSM shine.

FSM[S,D]

In order to use Akka FSM you need to mixin the FSM trait. The trait itself looks like this

trait FSM[S, D]

Where S is the state type, and D is the data type.

startWith

You can choose what state the FSM starts in by using the startWith method which has the following signature.

startWith(stateName: S, stateData: D, timeout: Timeout = None): Unit

initialize

Calling initialilze()performs the transition into the initial state and sets up timers (if required).

when

when is used to match the state, and is also used to control the movement to a new state using the inbuilt goto method, or possibly stay in the current state.

When uses pattern matching to match the events that a particular state can handle. As stated it is completely valid to stay in the current state or move to a new state

The examples that follow below will show you both stay and goto in action.

whenUnhandled

You should obviously try and make sure you cover all the correct state movements in response to all the events your FSM knows about. But Akka FSM also comes with the whenUnhandled method for catching events that were not handled by YOUR state handling (when) logic.

onTransition

You may also monitor the movement from one state to the next and run some code when this occurs. This is accomplished using the onTransition method.

onTransition has the following signature

nTransition(transitionHandler: TransitionHandler): Unit

Where TransitionHandler is really just a PartialFunction that has the following generic parameters

PartialFunction[(S, S), Unit]

Where the tuple is a tuple of “from state” to “to state”.

Time for an example

Lightswitch Example

This first example is a very simply FSM, that has 2 states, On and Off. It doesn’t really need any state, however the Akka FSM trait, always needs a Data object. So in this case we simple use a base trait for the Data which we don’t really care about.

The idea behind this example is that the lightswitch can move from Off –> On, and On –> Off.

This example also shows a stateTimeout in action, where by the On state will move from On, if left in that state for more than 1 second.

import akka.actor._
import scala.language.postfixOps
import scala.io.StdIn
object Demo extends App {
RunLightSwitchDemo
def RunLightSwitchDemo : Unit = {
//create the actor system
val system = ActorSystem("StateMachineSystem")
val lightSwitchActor =
system.actorOf(Props(classOf[LightSwitchActor]),
"demo-LightSwitch")
println("sending PowerOff, should be off already")
lightSwitchActor ! PowerOff
//akka is async allow it some time to pick up message
//from its mailbox
Thread.sleep(500)
println("sending PowerOn")
lightSwitchActor ! PowerOn
//akka is async allow it some time to pick up message
//from its mailbox
Thread.sleep(500)
println("sending PowerOff")
lightSwitchActor ! PowerOff
//akka is async allow it some time to pick up message
//from its mailbox
Thread.sleep(500)
println("sending PowerOn")
lightSwitchActor ! PowerOn
//akka is async allow it some time to pick up message
//from its mailbox
Thread.sleep(500)
println("sleep for a while to allow 'On' state to timeout")
Thread.sleep(2000)
StdIn.readLine()
//shutdown the actor system
system.terminate()
StdIn.readLine()
}
}

When run we get the following results

sending PowerOff, should be off already[WARN] [09/06/2016 07:21:28.864] [StateMachineSystem-akka.actor.default-dispatcher-4] [akka://StateMachineSystem/user/demo-LightSwitch] received unhandled request PowerOff in state Off/NoDatasending PowerOnMoved from Off to Onsending PowerOffMoved from On to Offsending PowerOnMoved from Off to Onsleep for a while to allow ‘On’ state to timeout‘On’ state timed out, moving to ‘Off’Moved from On to Off

There is something interesting here, which is that we see an Unhandled event. Why is this?

Well this is due to the fact that this demo FSM starts in the Off state, and we send a PowerOff event. This is not handled in the when for the Off state.

So if we apply this amended code, and run the demo again. We would now see this output instead

sending PowerOff, should be off alreadyalready offsending PowerOnMoved from Off to Onsending PowerOffMoved from On to Offsending PowerOnMoved from Off to Onsleep for a while to allow ‘On’ state to timeout‘On’ state timed out, moving to ‘Off’Moved from On to Off

Buncher Example

I have basically taken this one straight from the official Akka FSM docs.

This example shall receive and queue messages while they arrive in a burst and send them on to another target actor after the burst ended or a flush request is received.

Here is the full code for this example. It is a slightly fuller example, so this time we use a proper set of data objects for the states.

Where Can I Find The Code Examples?

In this post we will look at some of the core concepts around persistent actors. Which may seem strange since so far I have been saying that Akka is great cos it is lock free and each actor is stateless.

Truth is there are times where some sort immutable state, just can’t be avoided, and for that Akka provides us with the PersistentActor

Event Sourcing

Akka persistence does borrow a couple of ideas from other concepts out there in the wild, such as

Event sourcing

Possibly CQRS

Snapshots

if you have not heard of event sourcing before, it is fairly simple to explain. The idea is that you have a data structure that is able to receive events to set its internal state. At the same time you also have an event store that stores events in reverse chronological order.

These events are then played onto the object that accepts them, whereby the object will build its own internal state from the events played on to it.

The eagle eyed amongst you may be thinking won’t that lead to loads of events? Well yes it might.

There is however an optimization around this called “snapshots”. snapshots are a special type of event where they capture the object state as it is right now.

So in reality you would apply the latest snapshot, and then take any events that occurred after that and apply them, which would drastically reduce the amount of event play back that would need to occur against an object that would like to receive events from the event store.

I have written a fairly well received article about this before for the .NET space which also included working code and also includes the CQRS (Command Query Responsibility Segregation) part of it too. If you would like to know more about that you can read about it here:

Anyway I am kind of drifting of topic a bit here, the point is that Akka persistence borrows some of these ideas from other concepts/frameworks, so it may be of some use to have a read around some of it, in particular event sourcing and CQRS.

Dependencies

In order to work with Akka persistence you will need the following 2 SBT entries in your build.sbt file

Config

As with most things in Akka you are able to configure things either in code (by way of overriding things) or by configuration. I personally prefer configuration. So in order to get the persistence to work, one must provide the following configuration

There is an expectation there will be a receiveCommand implementation (this is within the EventSourced trait)

That the base trait is called EventSourced is also interesting

I personally find the fact that there is an expectation of a command and that we end up mixing in a trait that uses the name “EventSourced” to be quite a strong indicator of just how similar working with Akka persistence is to CQRS + traditional event sourcing ideas.

If we were to go further down the rabbit hole and keep looking into the base trait EventSourced we would see a couple more abstract methods that are of interest that are expected to be supplied by the end users code:

/**
* Recovery handler that receives persisted events during recovery. If a state snapshot
* has been captured and saved, this handler will receive a [[SnapshotOffer]] message
* followed by events that are younger than the offered snapshot.
*
* This handler must not have side-effects other than changing persistent actor state i.e. it
* should not perform actions that may fail, such as interacting with external services,
* for example.
*
* If there is a problem with recovering the state of the actor from the journal, the error
* will be logged and the actor will be stopped.
*
* @see [[Recovery]]
*/
def receiveRecover: Receive
/**
* Command handler. Typically validates commands against current state (and/or by
* communication with other actors). On successful validation, one or more events are
* derived from a command and these events are then persisted by calling `persist`.
*/
def receiveCommand: Receive

As stated these methods are abstract methods that YOUR code would need to supply to make the persistence actor stuff work properly. We will see more of this in a bit

Persistent Id

One of the rules you must follow when using persistent actors is that the persistent actor MUST have the same ID, even including across incarnations. This can be set using the persistenceId method as shown below

override def persistenceId = "demo-persistent-actor-1"

Snapshots

As I stated earlier snapshots can reduce the amount of extra events that need to replayed against the event source target (the persistent actor).

In order to save a snapshot the actor may call the saveSnapshot method.

If the snapshot succeeds the actor will receive a SaveSnapshotSuccess message

If the snapshot succeeds the actor will receive a SaveSnapshotFailure message

During the recovery process the persistent actor is offer a snapshotOffer from which it may restore its internal state.

After the snapshotOffer will come the newer (younger in Akka speak) events, which when replayed onto the persistent actor will get it to its final internal state.

Failure/Recovery

Special care must be taken when shutting down persistent actors from outside. For non persistence actor a PoisonPill may be used. This is not recommended for persistence actors due to how the commands are stashed until such a time as the journaling mechanism signals that things are stored. At which time the mailbox is drained. A better way is to use explicit shutdown messages

When we talked about mailboxes we discussed the fact that there were queues involves that are used as mailboxes.

Generally speaking where there are queues concerned there is also the possibility of dead letters.

In this post we will talk about what ‘dead letters’ are within Akka, and also look at how you can monitor for dead letters

What Are Dead Letters?

In Akka messages that can’t be delivered are routed to a synthetic actor which has the path “/deadLetters”. This is for NON transport lost messages.

Akka makes no guarantees for lost messages at the transport layer.

How Can You Monitor For Them?

Within Akka there is a concept of a event bus. That is a bus that Akka uses internally to send messages. You can also use this event bus for other purposes, such as publishing/subscribing to certain types of messages.

It is kind of like a topic based subscription system.

We can use this event bus to monitor for dead letters.

It is also worth noting that the event bus can be used a general pub/sub mechanism, you can read more about it here : Akka Event Bus

But for now let’s get back to the problem in hand which is how do we monitor for dead letters?

NOTE : According to the official Akka docs “Dead letters are not propagated over the network, if you want to collect them in one place you will have to subscribe one actor per network node and forward them manually.”

Are All Messages In DeadLetters A Problem

No, not all of the messages that end up in the deadLetters synthetic actor are problems. For example suppose an actor is sent several Terminate messages. Only one of these will take effect, the others will likely end up in deadLetters, but this is not a concern.

The deadLetters actor and the monitoring of it is more of a debugging aid than anything else. You will have to use some modicum of common sense when examining the results of the deadLetters logging, which we discussed above

Where Is The Code?

As previously stated all the code for this series will end up in this GitHub repo: