An interview with Mahesh Paolini-Subramanya (@dieswaytoofast)

Hi there! In this post we will learn something more about Mahesh Paolini-Subramanya. Mahesh is a well known developer and entrepreneur, very active in the Erlang community! Have fun reading his answers!

Interviewing the famous erlanger!

Mahesh – I’ve been a bit of a serial entrepreneur, bouncing around the tech world for around 25 years, with most if being Internet related stuff. (Seriously. Remember Gopher? I wasdoing stuff w/ Gopher and AFS back in ’92. And yes, it made sense at the time). I also have the – extremely dubious – honor of being involved in creating the first web/e-commerce system, the first java based financial services platform, and the first erlang-based cloud PBX, three products I may never, ever live down. Nowadays I’m a bit of a “full-spectrum” CTO, doing a bit of everything – Sales, Development, Marketing, Presentations, you name it. You can get the gory details here, if you so care. (By the way, dieswaytoofast is a gaming handle from the Quake days, for those who were wondering)

Paolo – Can you tell us how you you end up using Erlang?

Mahesh – I got involved with Java back in ’94, in the very early days of the dot-com era – applets seemed to be the greatest thing since the <blink> tag, and the future of web-sites as far as I was concerned. We developedAptela– one of the first hosted PBX services – back in ’99, and built it all on Java (remember, Java had not quite reached the “kitchen-sink-included” morass that it is today). Aptela evolved over the next 7 years into a fully-functional “cloud”-y environment, but as you might imagine, was also remarkably rube-goldberg-likein its complexity. It got to the point where a hugechunk of our development resources were focused on maintenance – tiny changes had ripple effects across the entire system, it took days to add something as simple as a drop-down box, etc.

Something had to change – and that something turned out to be everything. In 2006 we put existing product development on hold and rewroteeverything from scratch, in Erlang. This turned out to be one of the best decisions that we could have made 🙂

Paolo – What was the thing that intrigued you most about Erlang?

Mahesh – There are three different answers to the question, depending on the timeframe involved:

Scaling – When we started, it was all about the ability to scale – we had hit an absolute limit on scalability with our existing platform, and Erlang’s concurrency model seemed to be just the thing to help us break through our bottleneck

Testing – Shortly after we started development, we discovered that our productivity had gone through the roof. The combination of the actor model, functional component-ization, separation of concerns, immutable variables, and “side-effect isolation” gave us an unsurpassed ability to develop – and test! – our code efficiently. On our Java platform, a stack-trace was the start of a long nightmare, whilst in Erlang, it pretty much defined the solution (line-numbers have made this even better!)

Hot Code Loading – The ability to automagically roll in (and roll out!) changes to functions, modules, and heck system components, with our users being none the wiser was and is Ridiculously Awesome. We already were doing Hot Swapping in Java and given the aggressive grotesqueness of that mechanism, had discounted this feature in Erlang. Boy, were we wrong!

Incidentally, I wish I could point to one of the above and say “This is the greatest thing about Erlang”, but I can’t. IMHO, the elegant nature of the entire beast is what makes it great – the various attributes all fit together brilliantly, and the whole is indeed greater than the sum of the parts.

Documentation – Richard O’Keefe once said, “Erlang is designed for people who are willing to RTFM“, which may be true, but lord, does it need to be so hard? Every word tends to be important. Every word!. Mind you, this is a bit two-sided, in that once you bang your head against the wall, and eventually discover your problem, you inevitably find out that you mis-read the documentation. I’m not quite sure what the answer is here though 😦

Tooling – Emacs, while a spectacularly effective environment, is not the easiest environment to get acclimated to. Roberto Ostinelliis doing some wonderful stuff with SublimeErl, as isRicardo Jimenezwithvimerl, but – religious wars aside – they have quite a way to go before they get to the point where the entry-barrier is negligible.

(Real) Examples – Its all well and good to read about supervisors, but “real world” examples would be ludicrously useful. And this isn’t just about one_for_one vs simple_one_for_one. Its more about worker pools, supervisor hierarchies, etc within the context of an actual working example. People on erlang_questions are very helpful, its just that as a newbie, you probably don’t even know how to frame the question.

The commonality in the above answers is, in many ways, the main improvement need in Erlang, viz., they aren’t problems for experts, but are significant entry barriers for newbies. And, for a newbie, any friction is bad! Towards this,Fred Heberthas done yeoman’s service to Erlang with his excellent (!!!) Learn You Some Erlang, which I could not recommend higher – it has filled a critical hole in Erlang-space, viz,, an approachable, accessible, and most of all, fun book that is also immensely detailed and comprehensive.

Paolo – Erlang started as a programming language for the development of telephony applications. Among your specializations there are SIP and VoIP services, is Erlang really “so good” in these fields? Why?

Mahesh – VoIP services have been evolving over the last few years – they are now part of what is commonly called Cloud Communications, and covers the melding of synchronous (voice and video) and asynchronous (SMS, IM, Twitter, Facebook, G+, etc.) communications. For examples, think of a Skype call where you might have texted a link while talking to someone, or G+ hangouts with the ability to share embedded applications, or WoW with in-game chat and text – these are all examples of seamless multi-modal communication.

The point? These services are massively scaled, massively distributed, and massively reliable. Could you build these type of serves on Java/Ruby/Perl? Of course! But you could also build them in Assembler, if you were really felt masochistic!

Erlang, however, is designed from the ground up to build out exactly these kind of highly concurrent, web-scale systems and services – cloud communications just happens to be the specific use that we are putting this architecture to 🙂 So yes, Erlang is exactly “so good” for these type of applications!

Paolo – You are also famous for your knowledge of NoSQL databases. Can you point out what are the main tools Erlang related you used in your personal experience?

Mahesh – The tricky thing about massively concurrent shared-nothing web-scale systems is that you need massively concurrent shared-nothing web-scale persistence. This, or course, assumes that you do need persistence, and that you don’t need persistence bottlenecks. Given that in the end we are a telecom service, we’ve ended up with a lot of polyglot persistence. We’ve tried (and swapped out) quite a few NoSQL services, but eventually settled on CouchDB(BigCouch, to be precise) – admittedly because it was written in erlang 🙂

For most interactions, we use Benoit Chesneau‘s Couchbeamand ejson, along with Bob Ippolito‘skvc though we have quite a few proprietary services that we have written ourselves – primarily to do with service management as compared to database management.

On a side note, If we had it to do over again, we probably would have gone with Riak though – the scaling characteristics are much more appropriate for what we need, but thats a different story…

Paolo – I really appreciated your comments to a couple of my blog posts about gen_fsm. I know you love them as much as I do. Why do think so few Erlang developers know this behaviour in depth?

Mahesh – I suspect the main reason is that people equate Finite State Machines with Complexity, something that could not be further from the truth. Its not just protocols and locks (the common FSM examples). Au contraire, FSMs are – by definition – perfectly suited to tackle scenarios which lend themselves to existing in specific combinations of states or conditions – authentication, registration, workflow, etc. Its actually somewhat scary, once you start using them, you see them everywhere!

If anything, people should embrace them more – FSMs force you to think about your edge-cases, and that is the perfect way to prevent yourself from being woken up at 3am with the dreaded “System Down” phone call.

Paolo – Where is the Erlang community going? What can we expect in the Erlang future?

Mahesh – I do think Erlang is on the cusp of going mainstream. Polyglot development is becoming more common, and people (ok, some people!) are happily breaking out their systems into loosely coupled layers, e.g., Objective-C for the GUI, Ruby for the GUI server, and Erlang for the (distributed) infrastructure. I expect to see a lot more along those lines going forward, with the “one size fits all” camp coming along kicking and screaming.

As far as the community goes, the level of support one can get on IRC and erlang_questions is quite amazing, with most of the discussions tending to fall well on the good side of snarkiness 🙂 As the number of users grows, I am sure that the standard that has been set will be maintained.

Paolo – You were among the Erlang developers that took part to the spawnfest 2012. Can you tell us something more about this two days experience and about the project you implemented with your team mates?

Mahesh – I will be giving a talk about our experience atErlang Factory Lite Vancouver on July 28th. I urge all those who are interested to show up there – there is still time to go, and if you’ve never been to one before, it is a truly memorably experience!

I won’t spoil the talk, but I’ll just leave you with these few tidbits.

We called ourselves Team Net_Split, since the three of us were in three different countries – Argentina, the U.S., and Curacao! I myself was on a Scuba vacation in Curacao, and was coding between dives 🙂

We used the opportunity to start work on our new Project –Erlymob. It is an app that can be used to Crowd-source the sourcing of Crowds, i.e., allow Twitter users to create real-time flash-mobs, get to local-deals with little notice, promote activism etc.

We created just the bare-bones of Erlymob (github here), but as a pleasant and unexpected side-note, also ended up withapp-cache, which is an (evolving) tool for automatically setting up tables, caches, and sequences in mnesia.

There is a whole lot more – you’ll have to wait for the Vancouver presentation to find out the rest 🙂