Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Akka in Production: Our Story

Everyone in the Scala world is using or looking into using Akka for low-latency, scalable, distributed or concurrent systems. We want to share our story of developing and productionizing multiple Akka apps, including low-latency ingestion and real-time processing systems, and Spark-based applications.

When does one use actors vs futures?
Why did we go with Logback instead of Akka's built-in logging?
Can we use Akka with, or in place of, Storm?
How did we set up instrumentation and monitoring in production?
How does one use VisualVM to debug Akka apps in production?
What happens if the mailbox gets full?
What is our Akka stack like?
We will share best practices that we've discovered when building Akka and Scala apps, pitfalls and things we'd like to avoid, and a vision of where we would like to go for ideal Akka monitoring, instrumentation, and debugging facilities.

I tweaked the instrumentation traits to use composition instead of inheritance and work with become / FSM: https://gist.github.com/rawg/bdb02385b30cfb5c52b4. I explained the rationale at http://impliedconversion.wordpress.com/2014/07/30/composable-akka-metrics/.

Evan, I really, really, like this idea. Have you spent any time trying to get Stackable Traits to work with become/unbecome state transitions? Seems like the fact that the receive method is only implemented in the base ActorStack trait would break the stacking when receive behavior is changed with become.

Hi Archit, traditionally Akka hasn't been used in the transactional / guaranteed messaging space, as you have to build the ack'ing and at least once messaging on top yourself, and most people don't want to do that. But check out Akka Persistence, in 2.2.x or 2.3.0, which brings at-least-once (in fact exactly-once) along with message persistence and playback (resiliency even for process crashes).

6.
How we started using Scala
• Ooyala was a mostly Ruby company - even MR jobs
• Lesson - don’t use Ruby for big data
• Started exploring Scala for real-time analytics and MR
• Realized a 1-2 orders of magnitude performance boost
from Scala
• Today use Scala, Akka with Storm, Spark, MR,
Cassandra, all new big data pipelines
Saturday, October 19, 13

11.
Lessons Learned
• Still too complex -- would we want to get paged for this
system?
• Akka cluster in 2.1 was not ready for production (newer
2.2.x version is stable)
• Mixture of actors and futures for HTTP requests
became hard to grok
• Actors were much easier for most developers to
understand
Saturday, October 19, 13

27.
Flow control
• By default, actor mailboxes are unbounded
• Using bounded mailboxes
• When mailbox is full, messages go to DeadLetters
• mailbox-push-timeout-time: how long to wait
when mailbox is full
• Doesn’t work for distributed Akka systems!
• Real ﬂow control: pull, push with acks, etc.
• Works anywhere, but more work
Saturday, October 19, 13

28.
Flow control (Cont’d)
• A working ﬂow control system causes the rate of all
actor components to be in sync.
• Witness this message ﬂow rate graph of the start of
event processing:
Saturday, October 19, 13