Six Months of Akka.NET

I've been using Akka.NET in a real-world commercial project for a large organisation that operates in the finance industry. This project has been in development for 6 months now. I've wrapped up some of my thoughts on the technology / actor model in this post.

Conclusion

As Akka.NET is a framework, most people are interested in how it sits as a whole, before drilling into specifics. Much of what I've found can probably be applied to other actor model implementations such as Orleans.

I really liked the framework.

Much like my first foray's into F#, Akka.NET allows you to solve problems in a different way. Much of Akka.NET's benefits are 'publicised' arounds it clustering capabilities. And while these are certainly apparent and useful, it is the design patterns that you can create and use that is really what makes me excited about the actor model and this technology.

I did find it made automated unit testing harder, and its highly decoupled approach did have an impact on cohesiveness. It also comes with a large learning curve, particularly those new to actor model and unfamiliar with message-based or event-driven design patterns.

There is also some level of architectural / design investment in using the framework properly, so like any tool, you are going to want to use it wisely.

I wouldn't use Akka.NET on every project. And I'd certainly consider alternate actor model implementations based on the project. However, I definitely will continue to invest my time in the technology based on my experiences with it to date.

If you want to learn more about Akka.NET, I highly, highly, recommend Petabridge's free bootcamp.

Random thoughts

I'm not going to discuss all of the benefits of Akka, Akka.NET, and the actor model. There is plenty of content that can be found online for this. What I am going to talk about are some of the less obvious things that I discovered.

Decouple me

Akka.NET is highly decoupled.

Actors generally communicate or associate through an IActorRef (or equivilent). Which means you can only tell (fire and forget) or ask (request and response) them something. This IActorRef is a proxy, so you can't even directly access or cast to that actor, even if it exists in the same process.

Really great for testing, seperating concerns, and aggregating. Unfortunately, in larger codebases it can make the code less cohesive, and you often find yourself looking at message types in order to figure out the flow of the system. Not a deal-breaker with a sensible design though.

The double-edged concurrency model

One of the salient points of Akka.NET is its concurrency model.

Each Akka.NET actor has two messages queues in front of it (one for user messages, one for system messages). Akka.NET guarantees that an actor can only process one message at a time.

This simplifies application logic, because you no longer need to concern yourself with concurrency protection when executing code inside an actor. You can concentrate on the domain, and the business logic.

However, given that in Akka.NET everything is concurrent by default it does make automated testing more difficult. Thankfully Akka.NET comes with a TestKit, which makes unit tests much simpler to write, and more reliable. Unfortunately, in my experience, once you have a large suite of automated tests, you can sometimes get random test failures. This is more likely to occur in tests involving a scheduler (or TestScheduler) and in tests that are closer to integration tests, rather than unit tests.

Kids, this is why you should avoid writing code that that potentially spawns a background thread, assuming you want to test it of course.

Note: I've not yet invested enough time in trying to track down why we get these random failures, it might be that we need to change the dispatcher. But, in our experience, the problems with testing are not easy and apparent to fix.

Finite state machine all the things

It simplifies what would ordinarily be complicated actors, and makes it much easier to reason about their purpose and functionality.

One of the main ways we used this method is to make an actor available to receive messages, while it isn't ready to process them. We make a request to another actor to get the data that this actor needs in order to process messages. Any messages that aren't related to the loading of the actor are stashed, until the actor is ready to process messages. (Which is another call to Become, in order to switch the actors state)

One of my favourite features of Akka.NET.

Distributed computing is still hard

Okay, this one wasn't really a surprise.

I just wanted to point out than even though the tooling gives you a great framework to operate from, you are still going to have to be aware of, and consider the usual suspects, such as consensus, message delivery guarantees, network partitions, and any other sorts of failures.

You're also going to have to consider performance, particularly in regards to network IO. The Akka.NET team could write the best networking code in the world, but if your communication is very chatty or contains large message payloads, it ain't going to matter.

Location transparency is a lie

Okay, not entirely.

It is true that when all you have is an IActorRef to another actor you have no idea (nor should you care) if that actor exists in the same process, or on a different process. It still gives the same powerful scalability benefits, but there are a few less obvious gotcha's.

Actor selection only by default only works within the local actor system (i.e. same process). You can still do selection across a cluster with a grouped router, but it isn't as idiomatic.

You can use actor selection remotely, however, you need to know the remote host endpoint.

When creating actors through props, those props need to be serialisable.

You often consider the messaging and interaction between actors for performance reasons before potentially remotely deploying or communicating over the cluster.

State is no longer evil

Web applications are a staple of software development. HTTP is inherently stateless, however, applications are inherently stateful.

As developers we try to bridge this gap using constructs such as session state and cookies. In my view, these are hacks. They cause all sorts of problems, in terms of performance and scale.

Typical n-tier architecture and design revolves around using a database as a backing store to persist state. These constraints limit solution design, and introduce bottlenecks in the system.

This blog post talks about how one might implement stateful web applications with the actor model.

Design pattern are us

The actor model opens up some really cool design patterns, opening up many of the message-based patterns you can read about.

The ease at which you can implement patterns such as pub / sub and message aggregation patterns lets you solve problems in a different manner to how you probably typically would.

For example, in a typical application independent isolated HTTP requests might load a product with a particular identity. One of the patterns we came up was to aggregate these individual requests for data into one, to send to SQL. This limits the overhead on the database as one larger call is generally better performance than lots of small calls.

Actors have identity

Akka.NET has its own protocol, akka.tcp. Like the HTTP protocol that your browser knows how to interact with, the Akka.NET framework also knows how to interact with this protocol. You can communicate with actors across the cluster that you have not established any prior direct communication with.

It also means that you can host your entities on the actor system. Entities that store and track state, such as inventory, can be accessed with concurrency protection across the cluster. Fast in-memory access can modify that state, while writes to persistant storage can happen asynchronously as the response is sent back to the requestor.

Cluster it

Akka.NET's clustering is great. It works as advertised, and you have a lot of power and flexibility in how you deal with the cluster.

Unlike Orleans you do need to think a lot more about how you are going to distribute work and the messaging interaction as it involves less magic. Depending on the solution you are developing, this may, or may not, be a good thing.

There are a couple of tricky spots when you deal with clustering.

Sometimes the actor may not be ready to receive messages immediately after remotely deployment of an actor, so you may want to introduce some sort of acknowledgement message for when the actor is up and ready to accept messages.

Akka.NET's default serialisation is Protobuf for Akka.NET's well known message types, and JSON for your own application messages. Sometimes when there is a serialisation exception it is difficult to figure out which message type is causing the serialisation problem as you get a fairly vanilla stack trace from JSON.Net.