Archive

Hello! September 2013 has come and here I am once more to give you some information about another year of Erlang blogging 😀

Would you like to know something more about what I posted in this blog during 2013? Are you sure? Well, just keep on reading!

I can’t recall every single post I wrote this year, so I have to ad lib during this summary.

Since I noticed that you readers appreciate my interviews to famous Erlangers I kept on writing this kind of posts. I think nobody will be offended if I say that two of my favourite interviews this year were the one toKenji Rikitake and the one toSteve Vinoski. I obviously want to thank all the others Erlangers that accepted my request for an interview and I hope that you had the chance to read at least some of their inspiring ideas and thoughts!

I also promoted Erlang Conferences all around the world and I hope that many of you attended theEUC2013or the recentErlang Camp in Amsterdam (or the one that will be held soon in Nashville!).

Some posts were as usually more technical, in this sense I enjoyed very much writing the one aboutTDD and kata, the one aboutthe cost in terms of time you have in your supervisor when spawning multiple gen_servers. I also liked sharing with you the most interesting Erlang articles I read online lately in my recentpost.

That all! As always I want to thank all you guys for your support and patience. Hope you will keep on following my blog during these last months of 2013 and during 2014! Have a nice Erlang year!

Hello guys! Finally I had some free time to post on this blog :D. Today you can read my interview to Eugene Fooksman, a software developer at WhatsApp. As always I hope you will like it!

WhatsApp Eugene?

Paolo – Hello Eugene. Welcome to my blog! Would you like to briefly introduce yourself to our readers?

Eugene – Hello, thanks for inviting me. My name is Eugene Fooksman, I live in Silicon Valley, California, and I’m currently a software developer for the server group at WhatsApp, the company providing multi-platform mobile messaging service.

Paolo – Can you tell us something more about your experience with Erlang? When did you start with it? Why?

Eugene – I started using Erlang only here, at WhatsApp. My previous expertise and long time affection was C++, I used it for many years and, naturally, considered it to be the king of programming languages, as many C++ programmers do.

WhatsApp server is almost completely implemented in Erlang, so I had to learn it when I came here, with certain reluctancy at first, I must admit. But it took no more than about a week to completely fall in love with it.

Paolo – As an Erlang developer, what do you think about this coding language? Do you like it or is it just a part of your work?

Eugene – I like it a lot. Functional nature is a bit hard to accept for a person with object oriented mindset, but once you go over that line – the simplicity of many basic routine things never stops to amaze.

For instance, tail recursion being embedded in the very core and syntax of the language not only compensates for the lack of standard loops, but also totally changes the way you think of anything iterative in your code and provides very reach coding capabilities.

Another such example is pattern matching – very powerful way for processing heterogeneous values returned by the called code. It’s very disappointing for me now to not be able to do it in other languages.

Of course I’m not even talking about the real high-level advantages of Erlang as a perfect platform for concurrent programming and handling multi-user environment, like communication servers. Parallelism, transparency of remote vs local processes, and set of very well defined behaviors (which are essentially implementations of fundamental server concepts and patterns like generic sever, event manager, state machine etc.) – all of this makes writing server systems quite easy and elegant.

Paolo – In your opinion what are the fundamental things an Erlang developer should focus on in order to improve his coding efficiency?

Eugene – I would defi
nitely start with deep understanding of OTP principles and concepts. Supervision models can be a little tricky, but it’s worth spending time and effort figuring it out.

OTP has excessive set of tools and patterns, and it’s important to understand which ones are better for specific tasks. For instance, gen_server and gen_fsm can be two competing approaches for the same system. Another important thing is coding discipline and cleanness – something to always be utilized with languages without strong typisation, like Erlang.

Paolo – You are currently working at Whatsapp, can you describe in a nutshell the company?

Eugene – WhatsApp is a small big company.

We deliver billions of messages between millions of users, but at the same time we have only about 35 people working here, only about 20 of them are engineers, which include small mobile teams creating and maintaining client applications for 6 different smartphone platforms, and a server group, making sure all those messages are delivered. With the great team and very concentrated focus on our product – it’s a lot of fun to be here and help creating such a cool service.

Paolo – How long has Whatsapp been using Erlang? Did the company switch from a different language or did you decided to start the project in Erlang from the beginning?

Eugene – WhatsApp server has started from ejabberd – famous open source Jabber server written in Erlang. It was originally chosen because of a group of reasons, including openness, great reviews by developers, ease of start and the promise of Erlang’s long term suitability for large communication system. We started from ejabberd and made just few extensions and changes to get WhatsApp service up and running.

We have spent next few years re-writing and modifying quite a few parts of ejabberd, including switching from XMPP to internally developed protocol, restructuring the code base and redesigning some core components, and making lots of important modifications to Erlang VM to optimize server performance.

Paolo – According to this tweet, on December 2012 Whatsapp hit a new record: 7B msgs inbound, 11B msgs outbound = 18 billion total messages in one day. Do you think erlang helped to achieve such a wonderful results?

Eugene – We are delivering more than 7 billion message every day. It’s a big number, although it is not unheard of among internet and communication giants. Our great achievement is that we manage it with really small server footprint. And the consensus in our team is that it is largely because of Erlang. We’re managing to serve huge amount of connections from single front-end server (this is from last year’s Erlang Factory: https://twitter.com/igorclark/status/185871819427954688). I’m not sure these numbers can be easily matched with other technologies.

Paolo – Do you think you could achieve the same results with other languages? In your opinion, could the final result compete with the Erlang implementation in terms of time and lines of code?

Eugene – Yes, this volume of messages can be (and has been) achieved with other languages, but Erlang definitely makes it very easy, elegant and manageable by small team.

Paolo – As an experienced Erlang developer, what would you like to see in the future Erlang releases? Do you think there is still something missing in this language? Something that could potentially help spreading Erlang to all the developers out there?

Eugene – The are few things that could make writing code just a little easier. For instance, some concept of early returns from within functions and scopes. Also, assumed default values for ‘case’ and ‘if’ statements. Another thing is VM-level implementation of priority messaging for gen_server. Also we hope some of performance improvements to VM we’ve developed at WhatsApp can be adopted and used in future releases.

Hello everybody! Thanks to Erlang Solutions I have been able to interview Evan Miller, one of the speakers you can meet at the upcoming Erlang Factory which will be held in San Francisco (21-22 March) . Evan is mostly known as the author of Chicago Boss. Let’s know him better with this interview! Hope you will enjoy it!

Meet the boss!

Paolo – Hi Evan, it’s really nice to have you here. Can you briefly introduce yourself to our readers?

Evan – It’s great to be here, and I’m glad to do it. I’m a graduate student in economics living in Chicago who has a passion for programming. People in the Erlang world know me as the creator of Chicago Boss, the maintainer of ErlyDTL, and a loudmouthed defender of unpopular language features. Before grad school I briefly worked as an ops guy for Amazon.com and as a developer for IMVU, both out in Silicon Valley.

Paolo – How come a person with a degree in physics gets in touch with Erlang? Was it “love at first sight” or did you take a while in order to grasp it?

Evan – I stumbled across Erlang in a rather roundabout way. While I was involved in search operations at Amazon.com, I discovered that the source of almost all pain and suffering in the ops world can be traced directly to blocking, threaded socket code. This led me to tinker with Nginx — a non-blocking server — but although I love programming in C (really!), I couldn’t stand the callback-based API. Shortly thereafter, a friend of mine at IMVU introduced me to Erlang, which is non-blocking… but without the callbacks! I instantly saw through Erlang’s syntax warts and fell in love with “what’s on the inside” — the absolute best VM for doing any kind of network programming.

As for the physics connection, I agree with Joe Armstrong — the creator of Erlang and who incidentally is an ex-physics person — when he says that Erlang’s process model comes naturally to anyone who has studied relativity. Each process is its own “frame of reference”, in a manner of speaking — no process knows what’s on the inside of other processes, and heck, process clocks may be out of sync, but that’s OK! You learn how to program around it and end up with much more reliable systems than ones that assume there is a “true frame of reference” of some sort. There can’t be, because you have to assume that *any* server might fail or become unreachable in a networked environment.

Paolo – You are going to be one of the speakers at the next Erlang Factory held in San Francisco Bay area on March, what will be topic of your talk?

Paolo – Who should attend your talk? Do they need to be expert Erlang or Web programmers?

Evan – Everyone should attend! I’ll try to make it interesting for both experts and novices. For example, I’ll explain why Chicago Boss is so darned good at fast server-side template rendering, which is something that most languages have a lot of trouble with. I’ll also show some code samples to make things concrete.

Paolo – As the author of Chicago Boss would you like to introduce it briefly to our users?

Evan – Sure thing. Chicago Boss is a web framework that basically attempts to steal all of the good ideas from Ruby on Rails, and fix everything that’s broken: namely, slowwwwww templates, rampant RAM usage, lack of real-time messaging (WebSockets), and all this DevOps nonsense. So Chicago Boss combines fast templates with a clean API and a sophisticated “ORM”. I put that last bit in quotation marks because Erlang doesn’t really have objects, so we use a sort of fake object called a parameterized module that everyone on the Erlang mailing list hates with a passion, but which we love because it’s perfect for modeling data.

Anyway the framework is still in beta, but a lot of people are using it in production and we’re hoping to have a stable release soon!

Paolo – In your opinion, what are the main reasons that should make a web programmer switch from other platforms written in different languages as Rails, Symfony or Django to Chigago Boss?

Evan – Well, the first advantage is getting to work with a nice functional language like Erlang, even if CB sprinkles in some OO-style conventions here and there. Once you get the hang of it — which may take a while! — it’s generally easier to write correct code in Erlang than in non-functional languages.

But I think the real benefit actually comes down to operational costs — Chicago Boss runs smoothly on relatively little hardware and doesn’t need to be constantly restarted like most Ruby and Python servers. Erlang truly has a first-rate VM that is perfect for serving web pages. We’re really excited about running CB applications on ARM-based servers, which could really up-end the economics of server applications in the next few years.

Paolo – Chicago Boss in not the only option when talking about Erlang and web applications, what are the strength of Boss with respect to the others? Is there any feature you would like to “steal” from them?

Evan – CB is more of a full-stack framework than the others out there — in particular, it’s the only one that ships with any kind of database abstraction, letting you write your code first and choose from one of six SQL and NoSQL database options later. There’s also an email server baked in, and an internationalization system — we’ve really tried to provide a complete package, whereas most of the other Erlang servers tend to be more bare-bones toolkits.

Don’t tell my mother, but I never hesitate to steal! But most of what I’ve stolen has actually come from the world outside Erlang. For example, the default template language is a tag-for-tag reimplementation of Django’s template language, and there’s now experimental support for the Jade template language.

Paolo – What kind of data storage support may we have with Chicago Boss? Do you recommend any of them, and if so why?

Evan – As I mentioned there are many options — in the SQL camp there’s PostgreSQL and MySQL; in the NoSQL camp there’s MongoDB, Riak, and Tokyo Tyrant, and then there’s an Mnesia adapter which isn’t SQL per se but it is schema’d. I don’t want to start a religious war, so I’ll stay quiet about my own preferences — but it’s really easy to add support for more databases — usually just requires a couple hundred lines of code.

Paolo – At the beginning Boss was implemented around Misultin, but later you moved to Cowboy. Was it difficult to switch the architecture? How many code did you rewrite in that sense?

Evan – For a long time, Chicago Boss supported only Mochiweb and Misultin; as we were pondering how to support Cowboy’s alien-looking API, someone had the brilliant idea to write a Mochiweb-compatible wrapper around Cowboy! It’s called “mochicow” and lets us adapt CB to use Cowboy in rather short order. Anyway, most of the hard work here is actually done by the folks over at Nitrogen who created SimpleBridge, which is an abstraction layer over I think six different web servers. It’s really a nice little library and we’ve contributed code back to it.

Paolo – One strength of Chicago Boss consists in my opinion in the good amount and quality of its documentation. Would you like to address your favourite online resources to a developer willing to learn more about Chicago Boss? Is there any “real” book about Chicago Boss?

Evan – As much as I love killing trees, there’s no book yet! The closest thing to it is a PDF tutorial on the website, and people are always adding new code samples to the Wiki hosted on GitHub. The best way to get started is to work through the tutorial, peruse the API documents, and hop on the mailing list. It’s an active, friendly place where we welcome in newcomers all the time.

In 2011 I interviewed Francesco Cesarini for my blog. That was my second interview, a giant leap for my blogging experience. Today I am talking again with Francesco, mostly about his 2012 and an upcoming event organized by Erlang Solutions for all the erlanger in the USA: the Evening School of Erlang.

A 2013 full of Erlang!

Paolo – Hi Francesco! It’s good to have you here once again talking about Erlang. How was your Erlang year?

Francesco – 2012 has been a really good year. I spent a good part of it in the US, visiting companies using or looking to adopt Erlang to better understand their problems and needs. I taught the Concurrency Oriented Programming course at Oxford University both in the spring and the fall, alongside celebrating the 10th anniversary teaching at the IT University of Gothenburg. 600 served and counting. The best thing is seeing those who chose to work with Erlang after they graduated. Two EU funded research projects, Release and Paraphrase are sharing in excess of seven million euro. They have made excellent progress improving Erlang’s viability on multi-core architectures (we are talking tens of thousands of cores). A third project got funding in March and started in November. It is the Prowess project, taking property based testing to the next level. And Erlang Solutions is also doing very well. We expanded the offices in Krakow, moved to larger offices in London. There has been a huge interest for MongooseIM (our IM platform), our Erlang Management Console alongside lots of interesting courses and projects, all topped up and managed by great and dedicated staff. Record numbers at the San Francisco Erlang Factory and the Erlang User Conference, a lot more Erlang Factory Lites compared to 2011. It is clear that the adoption of Erlang is growing faster than ever before. I can’t wait for 2013.

Paolo – During the year many Erlang related events are held all around the world. In your opinion is this the best way to spread the Erlang word? How can we all help?

Francesco – The whole community is doing a fantastic job, so more of everything which builds the community is probably the right answer. We need more bloggers sharing their experiences, more people speaking at conferences and writing articles about their experiences. We need people to run user groups, self study circles, coding dojos and katas. We need better tools catering for the 21st century. Think cloud, embedded, massive scalability, with tens of thousands of nodes all interacting with each other. We need more research, more universities using Erlang as a tool to teach aspects of computer science such as distribution, concurrency and multi-core programming. And most important, we need to start speaking (and learn how to speak) to the CXO level. They are the decision makers who sit on the budgets and have a say on how and if Erlang is used out there. It is a very different language than that spoken by engineers. Instead of talking about cool debugging techniques or getting embroiled in the frames debate, we should be taking about financial benefits Erlang brings. How one reduces development, operations and hardware costs, making the software future proof towards future multi-core architectures.

Paolo – If you ask around you may find that many companies are afraid about using Erlang, mostly because they have a lot of know-how only about mainstream languages and fear the switch. How can we tackle this problem?

Francesco –I recently sat in a meeting where they were about to start a project with 100 Java developers. We suggested running a project in parallel, where we said we could do the work in half the time with 10 Erlang developers. It was a switching problem which had to handle peaks of traffic; Erlang was the perfect fit. The answer we got was, “If you are right, what do we do with our Java team”! This is fear, not common sense.

One has to introduce Erlang (or any emerging technology) starting small. Do a prototype or a proof of concept, show the advantages, and build on success. Mike Williams, co inventor of Erlang, include design by prototyping: In is not good enough to have ideas, you must be able to implement them and show they work. And make mistakes on a small scale, not in a full production project. When moving away from mainstream, at least in large companies, the conservative naysayers will be circling over you like vultures, waiting for the slightest hint of trouble. If you are 10% above budget, it does not matter if your budget is 1/10 of similar projects using mainstream languages. You are still over budget! Don’t give them the opportunity. But I am not answering your question, am I? My advise is to find your Erlang champion within the organization, make sure you are using the right tool for the job, get management buy in and start small.

Paolo – As many other Erlang developers, I am a member of the Erlang Questions mailing list; lately I spotted there a message from Erlang Solutions about an event called “Evening School of Erlang”. Can you tell us something more about it (e.g. locations and prices)?

Francesco –One of the biggest hurdles to adoption is lack of programmers. That was a clear concern when visiting companies on the US West coast. There are few experienced Erlang programmers commanding extremely high salaries and often changing jobs when new opportunities arise. To jump start the Erlang job ecosystem in the US, we decided to try out the Evening School of Erlang. Two nights a week, for four weeks, local trainers will be giving Erlang classes to developers looking for a career change. A few weeks after the course, those attending will be offered to take the certification exam. These classes will be heavily subsidized by corporate sponsors looking at recruiting/placing Erlang developers and will cost 199 USD. In exchange, your contact details will be shared with these sponsors.

In the new year, we are starting the courses on the 8th of Jan in San Francisco, close to Union Square, followed by Pasadena (Feb 19th) and San Jose (Feb 20th). Mountain View and Santa Monica will be announced soon. The goal for 2013 is to train 400 developers while expanding the network to cover Chicago, Seattle, Boston, New York, DC, and any other city where companies recruiting are willing to sponsor.

For those not interested in changing jobs but interested in learning Erlang, there is a corporate rate available which reflects the actual cost of the class. In this case, the details of the attendee will not be shared.

Paolo – Who may want to attend this kind of Erlang course? What are the topics and what is the Erlang level one must have in order to fully grasp them?

Francesco – Everyone who is interested in a career change and believes Erlang is the way forward. While we are hoping to attract experienced developers, students are just as welcome. Topics include:

Introduction

Basic Erlang

Sequential Programming

Concurrent Programming

Process Design Patterns

Process Error Handling

Code Updating

Records and Funs

ETS tables

Distributed Programming

Advanced Constructs

Ports and Sockets

Style and Efficiency

Paolo – In the aforesaid Erlang mailing list message, there was some reference to a “Certification Exam”. Can you tell our readers what it is? Will the exam price be included in the course price? I guess Erlang Solutions will the entity that grants the certification, am I right?

Francesco –The Certification in Erlang provides validation of Erlang/OTP knowledge. We provide two exams, one for Erlang and one for OTP. At a corporate level this validation is important both for employer and employee. It allows for employers to identify potential employees who are suitably skilled, and provides an on-going upgrade path to suit an organisation’s staff development plans and career aspirations.

It is possible to sit the examination for certification directly following the class, but we recommend that candidates undertake some self-study and obtain work in addition to the course, prior to sitting the exam. It is not easy to pass.

The Exam for the San Francisco class will take place in conjunction with the Erlang Factory in late March. The cost is included in the cost of the class and is subsidized by the sponsors.

Paolo – Do you think this kind of course may be helpful in order to get an Erlang related job? How can one developer attending to course to get in touch with these companies?

Francesco – The Evening School of Erlang is being sponsored by companies looking to recruit Erlang developers. At the end of every class, sponsors will give a 5 minute presentation on what they do with Erlang and pay for a round of drinks. These companies will get in touch with you.

Paolo – How can company become a supporter for this kind of event? How to get more information about it?

Francesco – For this and other classes, we are looking for support in the following areas:

Help in spreading the word to friends, colleagues and communities you believe might be interested in attending

Sponsors interested in recruiting Erlang developers willing to help out cover the costs of the trainer and material

Trainers interested in teaching the course

Companies with training facilities all across the US seating 10 – 20 delegates

You are welcome to contact me via our website, where you can also find more information and register:

Paolo – What about European Erlang developers? As far as I know you already started some courses in Krakow. How long should we wait for something like this to happen all around the Europe (maybe Italy :D)?

Francesco –This is entirely driven by market needs. Europe has been teaching Erlang in its universities for over 20 years! Thousands of developers have been trained, so the situation is not as acute as in the US. But if companies come to us and say they are having a hard time recruiting, we will certainly consider it.

You mention Krakow, where Erlang Solutions had to recruit. We tried out the evening school of Erlang back in 2008. The first time, 7 people registered. The second time, the number went up to 20, and the following year, we had 70 applicants. Word got around. The universities there started teaching it. They started doing research. And the yearly Erlang Factory Lite has for the last three years attracted in excess of a hundred attendees.

Paolo – I do believe 2012 was a good year for Erlang: a lot of events, courses and some new good book. Our world proved to be fault-tolerant to 21st december 2012, so What should be expect from 2013?

Francesco – Massive concurrent expansion of the language adoption. Lots of books are being written. I am co-authoring the follow up of “Erlang Programming”, tentatively called “Designing for Scalability with Erlang/OTP”. It will be published by O’Reilly. Joe Armstrong is working on the second edition of his book. On top of that, there are four other books in the writing which I am aware of but am unsure if I am allowed to speak about. My wish of having a whole bookshelf with books on Erlang will be realized this year. We are planing at least an Erlang Factory Lite per month, break the previous attendance records at the Erlang Factory in San Francisco and Stockholm and continue our work on educating and helping people. We will soon be launching a new community site aimed at replacing Trapexit. On top of that, Erlang Solutions will continue its international expansion and help grow the community. So all in all, if 2012 was an excellent year, we are really looking forward to working with the community in making 2013 an even better one!

Hi there! Today you can read my interview to Fred Hebert. For those of you living on the moon for the last couple of years, he is the author of “Learn You Some Erlang for great good!”. In the interview we talked about Erlang, his book and distributed systems in general. Hope you will enjoy it as much as I did.

Answer you some questions for our good!

Fred – I’m Fred Hebert. I’m a Québécois/French Canadian, and I’m a developer/programmer by accident, mostly self-taught. I’m mostly known in the Erlang community for having written Learn You Some Erlang for great good! I’ve spoken in a few conferences, spent a year training people in Erlang and writing course material. More recently, I’ve won Erlang User of the Year 2012, which I’m very happy about.

Other than that, I’m mostly unknown and plain. I’ve played bass guitar for years, although my work with LYSE has made me give it up in a large part the last two years. I’ve done improv, love to read, and so on. I’m rather cynical and verbose, I enjoy science fiction, music, documentaries, and watching Hockey. I’m studying part time in a local University (computer science, the equivalent of a minor) while working full time for BLOOM Digital Platforms, doing Real Time Bidding with Erlang. See http://ferd.ca/rtb-where-erlang-blooms.html for an explanation.

Paolo – What is your IT background and when did you start using Erlang?

Fred – This is a bit of a long story. I’ve never really known what I wanted to do in life. I mostly am driven by whatever I find interesting, in some kind of balancing act. At different parts of my life I’ve wanted to be a radio announcer, comedian, writer, musician, scientist (whatever kids would perceive as science), and so on. I decided to go in Multimedia, given it appeared to be a good way to be creative; I’d do web design and get paid for it, opposed to my views of radio, comedy, or music, where I’d work, but not really get paid for it.

The multimedia course introduced me to programming with basic JScript and PHP classes. That was an entirely new world to me, and I loved it. I thus started focusing on that, learning at night, until I landed an intership at a local dating site, the largest one in Quebec, with millions of members and thousands of them active at any point in time. I learned a lot there related to business processes, dealing with people, maintaining older software, refactoring, some scaling principles, and a lot more.

I had a good attitute, loved to learn and explore, try new languages, and this landed me a place in a tiny experimental Research and Development department. This was a small team with 3 people, one of which was on the company’s board. Because he was on the team, we were on the same level as the full executive board in the company’s hierarchy. The board member gave me the assignment to look into Erlang given Facebook were using it for their chat system and we were looking to rebuild one. Then, the board member left the company for unrelated reasons, and the other guy left on parental leave.

That left me alone as one-man department for 3 months, with nobody in the company able to tell me what to do because I was in an organizational black hole. I took these three months and a few books to teach myself Erlang. I was hooked on the concurrency model; it just fit in very nicely with the way I think. Eventually the board figured out my job was absurd, and demoted me back to integrator. I got bored and wanted out. I had no qualifications whatsoever, no paper to prove I was competent to be a real programmer, so I did two things: I started going to university part time, and I started writing LYSE.

I had a few chapters done and my name started to get out, so one day I sent Francesco Cesarini a message on twitter asking him if he needed anybody from Canada. He didn’t really need one, but they gave me a spot as the training course developer at Erlang Solutions Ltd. I was also to train people with the material I wrote for them. I did it for a year, enjoyed it a whole lot, but missed developing stuff. That’s when I got an offer to work at BLOOM Digital Platforms, and now I’ve been there for almost a year.

It’s been a large mix of luck and lots of hard work. That’s my background.

Paolo – You are some kind of celebrity among Erlang developers, mostly because of your book (which I find very inspiring). Why did you decide to write it?

Fred – As mentionned earlier, part of it was that I wanted to fix a few issues I had with Erlang while learning it, namely not really having a good online resource for free for it.

I began growing the idea of writing this after reading Miran Lipovača’s Learn You a Haskell for great Good! (LYAH) tutorial; I thought he did a great job with it. As I already knew him from online forums, I asked him how he felt about me writing an Erlang counterpart to his book. He liked the idea, being somewhat interested in Erlang.

A more complete list of reasons would be that I wanted to prove I knew stuff to get better jobs, I wanted to force myself to learn Erlang better, I wanted to know what writing a book would be like (I wrote plenty of bad tutorials before), needed some kind of programming-related project, wanted to help the Erlang userbase grow (so I could get jobs in it) and who knows, maybe I could get a dead tree book out of it, which sounded incredibly cool to me. Now it’s somewhat more normal, but things tend to look impressive and impossible up until you actually try to do them. Then it tends to become simpler as you learn how it’s done.

Paolo – How come did you write a free internet book instead of a “normal” book? I actually bet most of the erlangers out there would be willing to pay you some beers for this 🙂

Fred – Because one of the reasons I wanted to write LYSE was to make it less painful to learn Erlang (no need to pay), it has been imperative for me to keep an online version for free. Writing the website version first would give me more leverage on that point if a publisher ever wanted to ask me to turn it in a book, as they did for LYAH.

My original knowledge of Erlang was limited, and I’m also rather bad at keeping focus on a single task for a long time. Having a public, incremental development process allowed me to get to get more reviews, and it kept me from giving up as it would make me look dumb.

That was a lot of pressure. Last of all, I don’t think I could get a book deal with my experience at the time without going public. Making things online for free and in a format that was alright for books just gave me a chance that at some point, someone would think “you know what? let’s get this printed” and hand me a contract. Which pretty much happened. It was a safe way out for me.

Regarding beers, I do not drink. I will however recommend that people offer beers to people who helped me a lot — reviewers and proof-readers. They’re in the FAQ (http://learnyousomeerlang.com/faq) for the site. I benefited a lot from LYSE: job offers, reputation, traveling, and friends. The reviewers are doing a very thankless and selfless work compared to me, and they deserve many thanks and beers.

Paolo – Which chapters were the most difficult to write for you as an Erlang expert? And which ones do you think are the most difficult to understand for an Erlang newbie?

Fred – I didn’t really consider me an Erlang expert until people started calling me that way. I always hated calling myself an expert because I invariably become full of myself, and I risk turning into a jerk, someone I don’t want to be. For most of the chapters, I had to do a big part of discovery on my own, although I did have a lot of help from the community at times.

There were 3 chapters I found particularly challenging to write. The first one is the one on exceptions — the one where I explain the difference between errors, exits, and throws. The challenge there comes from the fact the semantics for these 3 types of exceptions are loosely defined. I asked Robert Virding (who helped create Erlang) and Richard Calrsson (who added try … catch to the language) what the meanings were, and they didn’t agree. I figured out the semantics weren’t very clear, so I tried to hit a middle ground, extract what I read from the Erlang libraries, and push for a clearly defined difference in the chapter.

The second and third ones (the most difficult ones) were the chapters on releases and relups. That stuff is somewhat nasty. There are two concurrent systems, with very few people understanding them very well (this is getting better though). Their documentation seems to be coming straight from the halls of Ericsson and are hard to get into. People have a tendency to use rebar and blindly take templates without really understanding the systems behind it. I had a lot of decyphering, a lot of experimenting, and a lot of banging-my-head-against-a-wall to do about them. I’m glad I did it, though, I think it needed to be done.

The most difficult to understand for newbies are probably the Common Test and Mnesia chapters. They’re going to be harder to grasp and they’re long pieces of work. Oh and the FSM one, with the asynchronous trading system. The protocol is hard to test, it’s rather complex, and it’s another long chapter.

Otherwise, I’d guess all the chapters on concurrency are challenging because I fully expect any newcomer to skip over all the basics and jump in there, then get confused with all the weird features I assume they know by that point. They ignore them until I add more and more and finally it no longer makes sense. It’s a funny thing, and I don’t think it can be avoided. I’d probably do the same.

Paolo – I like the way you made it easy for people to understand a topic using different skeletons of “real life applications” for each chapter. How come did you follow this approach for your book? Was it difficult to link chapters to application?

Fred – It’s the one that always felt the best for me. As I said before, I’m mostly self-taught, and that means I can get to bang my head on the wall a lot when it comes to taking the examples in books and applying them in real life. I also hate “exercises left to the reader”. I tried to improve things by doing something I felt would have been better and that’s what it gave as a result. In the end, it’s nothing more than solving problems the way they would please me. It just happens that lot of other people have similar demands.

Of course, even the book examples are not enough to get by in real production setting — I’m constantly reminded of that by reader questions. It’s somewhat unavoidable unless you’re ready to put even more time in it. At some point you get a bit bored and want to move on, so you leave holes here and there.

Then there will always be people who hate my way of writing or teaching. That’s not avoidable. Fortunately there’s a whole lot of other books to help them. The more variety we have, the easier it is for everyone to learn in the way that’s most comfortable to them.

Paolo – Let’s change topic now. Among your specialities, there are also web developement, javascript and PHP. What are your favourite Erlang tools for connecting web front end interfaces to Erlang back end?

Fred – For web servers, I tend to go with cowboy.

Generally that’s a tough question, though. I started with web stuff, experimented with Erlang for the web a bit, but I do very little web development with it. I prefer to sit in the dark rooms of backend development, where my customer is another developer rather than a non-technical user. They act like a buffer between me and non-technical users, for whom what I do tends to look like black magic and I’m finding myself isolated from spec changes, design by committee, etc. far more than others. It’s easier to build on stable ground.

Being a front-end dev, or a dev that works close to the front-end is a thankless, difficult job where you juggle countless pieces of technology, bad protocols, people (who are always full of good or bad surprises), users with all kinds of software and weird behaviours. I try to stay away from it because it’s easier on my morale and keeps me from going entirely cynical. Props to people who love it and thrive in that environment, they’re impressive.

I’m not a fan of most web frameworks in Erlang right now, especially for larger websites. They’re almost all taking the monolythic approach of Ruby on Rails or Django where you develop things inside the framework.

The way I think web development should go is to write your entire application in Erlang, testable and independent from the web, and then plug it in a web front-end, or a framework that is almost just middleware for your app. That should make testing it easier, and extending it for more interactions also easier (say, mobile apps or non-HTTP front-ends).

We’ve got this rich framework in place called OTP, and a language that’s fantastic to handle everything server-side. We’ve got a piece of technology that should be great for these kinds of tasks. Then we take it all and shove it inside a framework inspired by what is done in other languages without the capabilities of Erlang for server-side stuff. I feel like using a monolythic framework that forces me to shove my design in there is restrictive. OTP is my framework. Web frameworks are tools to help me put my OTP stuff online.

I think we’ve got it kind of backwards, but I have no data to prove my ideal way would be better. It’s all speculation.

Paolo – You where among the partecipants of Spawfest 2012. Can you tell us something more about your project?

Fred – Yeah, the project is The Erl Next Door. I’ve worked on it with Malcolm Matalka. We wanted The Erl Next Door (tend) to be easy as hell to use for tutorial writers and people who want to demonstrate their libraries and applications on their own website.

We were tired of people having to explain how to install stuff. People always want a gem-like system for Erlang and we tried plenty of times as a community, but nothing stuck. We decided to explore things the way PLT Scheme (now Racket) did it with PLaneT (http://docs.racket-lang.org/planet/Using_PLaneT.html), something you can use inside your code and fetch applications dynamically.

We did make a couple of changes to how PLaneT works — we wanted something for tutorial writers and users first. We made it so any .erl or .zip file can be used to download a module or an OTP application, and get them built automatically no matter what tool they use. Then they’re loaded in memory and can be reused across sessions, although you can specify to dump the files in /tmp/ or something like that to make them impermanent.

Tutorial writers can just link to a zip or erl file anywhere in their code by adding the rel=”erlang-tend” attribute to hyperlinks to such files, or a tag in the header of the page. TEND can then scrape the page, find all code mentioned, and install it in one shot. I could then make Learn You Some Erlang have all its code downloaded in a function call (tend:load(“http://learnyousomeerlang.com/&#8221;)), or temas that have a given product could let you try it by doing tend:load(“path-to-my-tutorial’s-code”). You could make yourself a “web development bundle” for example, just by listing libraries in a webpage and sending TEND there.

A funny side-effect of making it loosely-specified like that is that it becomes rather trivial to write a repository system (like PLaneT) using this. For example, github and bitbucket both allow to download .zip files of any tags or the master/default branch by default. You could make your own repository while hosting no files, just linking to different sources.

As soon as I’m done with LYSE, I’ll be putting a bit of time into making a full blown repository for Erlang apps with a wrapper to make it friendly, entering the already overcrowded arena of dependency management in Erlang. Maybe this time it can stick. Worst case scenario is we have another bit of code nobody adopts. The idea of bundles have been left somewhat unexplored so far. Maybe that’s where I can make it fit in.

Paolo – What books, blogs and so on would you recommend to a young developer who wants to learn more about distributed systems architecture and development?

Fred – Good question. There aren’t many I know of. I stole a copy of the old “Concurrent Programming in ERLANG” (second edition) from my first job; the books are worth a couple hundreds of dollars now and they didn’t know they had it. The first part is free on erlang.org, but the best part is the second one. It shows a lot of the guiding principles behind Erlang’s distributed libraries, and it’s a great source for people who know programming, understand a bit of Erlang, but don’t know much about distributed programming.

I would then recommend Amazon’s Dynamo paper. It’s an incredibly good paper to condense a lot of material and compromise in very few pages. It shows a crapload of great stuff.

Reading on “The Fallacies of Distributed Computing” is also a great idea.

I tried to synthesize all of that content in my Distribunomicon to make it easier to understand, but getting an intuitive understanding of it in any way at all is good.

Then, I’d direct people to try and understand vector clocks/Lamport clocks. They’re a great way to think about how time is relative on different machines or processes, and it helps gain a more intuitive understanding of things.

Once that is somewhat well understood (CAP is a theorem, not something you can ‘beat’, there are gradients of consistency, it’s not either CA or CP with no middle ground), looking into PACELC is interesting. It’s basically the cap theorem, extended. PAC is “during a (P)artition, do you pick (A)vailability or (C)onsistency”, and “(E)lse, do you pick (L)atency or (C)onsistency”. For this one, http://www.slideshare.net/abadid/cap-pacelc-and-determinism and http://dbmsmusings.blogspot.ca/2010/04/problems-with-cap-and-yahoos-little.html are decent introductions. Exploring PACELC is getting into more and more obscure parts of distributed system theory and you’ll find yourself having more trouble discussing it with people, but it always helps to know about it.

Paolo – Anything else?

Fred – Yeah, I’d like to announce that Learn You Some Erlang is now available for preorder at http://nostarch.com/erlang and on sites like O’Reilly’s store or Amazon. The paper book is in black and white, and the ebooks are in color.

No Starch Press have been pretty flexible with most of my demands and they’ve been fine with allowing a copy of LYSE to remain online. They’re very open people and understand new technology better than many publishers, in my opinion.

Hi there! In this post we will learn something more about Mahesh Paolini-Subramanya. Mahesh is a well known developer and entrepreneur, very active in the Erlang community! Have fun reading his answers!

Interviewing the famous erlanger!

Mahesh – I’ve been a bit of a serial entrepreneur, bouncing around the tech world for around 25 years, with most if being Internet related stuff. (Seriously. Remember Gopher? I wasdoing stuff w/ Gopher and AFS back in ’92. And yes, it made sense at the time). I also have the – extremely dubious – honor of being involved in creating the first web/e-commerce system, the first java based financial services platform, and the first erlang-based cloud PBX, three products I may never, ever live down. Nowadays I’m a bit of a “full-spectrum” CTO, doing a bit of everything – Sales, Development, Marketing, Presentations, you name it. You can get the gory details here, if you so care. (By the way, dieswaytoofast is a gaming handle from the Quake days, for those who were wondering)

Paolo – Can you tell us how you you end up using Erlang?

Mahesh – I got involved with Java back in ’94, in the very early days of the dot-com era – applets seemed to be the greatest thing since the <blink> tag, and the future of web-sites as far as I was concerned. We developedAptela– one of the first hosted PBX services – back in ’99, and built it all on Java (remember, Java had not quite reached the “kitchen-sink-included” morass that it is today). Aptela evolved over the next 7 years into a fully-functional “cloud”-y environment, but as you might imagine, was also remarkably rube-goldberg-likein its complexity. It got to the point where a hugechunk of our development resources were focused on maintenance – tiny changes had ripple effects across the entire system, it took days to add something as simple as a drop-down box, etc.

Something had to change – and that something turned out to be everything. In 2006 we put existing product development on hold and rewroteeverything from scratch, in Erlang. This turned out to be one of the best decisions that we could have made 🙂

Paolo – What was the thing that intrigued you most about Erlang?

Mahesh – There are three different answers to the question, depending on the timeframe involved:

Scaling – When we started, it was all about the ability to scale – we had hit an absolute limit on scalability with our existing platform, and Erlang’s concurrency model seemed to be just the thing to help us break through our bottleneck

Testing – Shortly after we started development, we discovered that our productivity had gone through the roof. The combination of the actor model, functional component-ization, separation of concerns, immutable variables, and “side-effect isolation” gave us an unsurpassed ability to develop – and test! – our code efficiently. On our Java platform, a stack-trace was the start of a long nightmare, whilst in Erlang, it pretty much defined the solution (line-numbers have made this even better!)

Hot Code Loading – The ability to automagically roll in (and roll out!) changes to functions, modules, and heck system components, with our users being none the wiser was and is Ridiculously Awesome. We already were doing Hot Swapping in Java and given the aggressive grotesqueness of that mechanism, had discounted this feature in Erlang. Boy, were we wrong!

Incidentally, I wish I could point to one of the above and say “This is the greatest thing about Erlang”, but I can’t. IMHO, the elegant nature of the entire beast is what makes it great – the various attributes all fit together brilliantly, and the whole is indeed greater than the sum of the parts.

Documentation – Richard O’Keefe once said, “Erlang is designed for people who are willing to RTFM“, which may be true, but lord, does it need to be so hard? Every word tends to be important. Every word!. Mind you, this is a bit two-sided, in that once you bang your head against the wall, and eventually discover your problem, you inevitably find out that you mis-read the documentation. I’m not quite sure what the answer is here though 😦

Tooling – Emacs, while a spectacularly effective environment, is not the easiest environment to get acclimated to. Roberto Ostinelliis doing some wonderful stuff with SublimeErl, as isRicardo Jimenezwithvimerl, but – religious wars aside – they have quite a way to go before they get to the point where the entry-barrier is negligible.

(Real) Examples – Its all well and good to read about supervisors, but “real world” examples would be ludicrously useful. And this isn’t just about one_for_one vs simple_one_for_one. Its more about worker pools, supervisor hierarchies, etc within the context of an actual working example. People on erlang_questions are very helpful, its just that as a newbie, you probably don’t even know how to frame the question.

The commonality in the above answers is, in many ways, the main improvement need in Erlang, viz., they aren’t problems for experts, but are significant entry barriers for newbies. And, for a newbie, any friction is bad! Towards this,Fred Heberthas done yeoman’s service to Erlang with his excellent (!!!) Learn You Some Erlang, which I could not recommend higher – it has filled a critical hole in Erlang-space, viz,, an approachable, accessible, and most of all, fun book that is also immensely detailed and comprehensive.

Paolo – Erlang started as a programming language for the development of telephony applications. Among your specializations there are SIP and VoIP services, is Erlang really “so good” in these fields? Why?

Mahesh – VoIP services have been evolving over the last few years – they are now part of what is commonly called Cloud Communications, and covers the melding of synchronous (voice and video) and asynchronous (SMS, IM, Twitter, Facebook, G+, etc.) communications. For examples, think of a Skype call where you might have texted a link while talking to someone, or G+ hangouts with the ability to share embedded applications, or WoW with in-game chat and text – these are all examples of seamless multi-modal communication.

The point? These services are massively scaled, massively distributed, and massively reliable. Could you build these type of serves on Java/Ruby/Perl? Of course! But you could also build them in Assembler, if you were really felt masochistic!

Erlang, however, is designed from the ground up to build out exactly these kind of highly concurrent, web-scale systems and services – cloud communications just happens to be the specific use that we are putting this architecture to 🙂 So yes, Erlang is exactly “so good” for these type of applications!

Paolo – You are also famous for your knowledge of NoSQL databases. Can you point out what are the main tools Erlang related you used in your personal experience?

Mahesh – The tricky thing about massively concurrent shared-nothing web-scale systems is that you need massively concurrent shared-nothing web-scale persistence. This, or course, assumes that you do need persistence, and that you don’t need persistence bottlenecks. Given that in the end we are a telecom service, we’ve ended up with a lot of polyglot persistence. We’ve tried (and swapped out) quite a few NoSQL services, but eventually settled on CouchDB(BigCouch, to be precise) – admittedly because it was written in erlang 🙂

For most interactions, we use Benoit Chesneau‘s Couchbeamand ejson, along with Bob Ippolito‘skvc though we have quite a few proprietary services that we have written ourselves – primarily to do with service management as compared to database management.

On a side note, If we had it to do over again, we probably would have gone with Riak though – the scaling characteristics are much more appropriate for what we need, but thats a different story…

Paolo – I really appreciated your comments to a couple of my blog posts about gen_fsm. I know you love them as much as I do. Why do think so few Erlang developers know this behaviour in depth?

Mahesh – I suspect the main reason is that people equate Finite State Machines with Complexity, something that could not be further from the truth. Its not just protocols and locks (the common FSM examples). Au contraire, FSMs are – by definition – perfectly suited to tackle scenarios which lend themselves to existing in specific combinations of states or conditions – authentication, registration, workflow, etc. Its actually somewhat scary, once you start using them, you see them everywhere!

If anything, people should embrace them more – FSMs force you to think about your edge-cases, and that is the perfect way to prevent yourself from being woken up at 3am with the dreaded “System Down” phone call.

Paolo – Where is the Erlang community going? What can we expect in the Erlang future?

Mahesh – I do think Erlang is on the cusp of going mainstream. Polyglot development is becoming more common, and people (ok, some people!) are happily breaking out their systems into loosely coupled layers, e.g., Objective-C for the GUI, Ruby for the GUI server, and Erlang for the (distributed) infrastructure. I expect to see a lot more along those lines going forward, with the “one size fits all” camp coming along kicking and screaming.

As far as the community goes, the level of support one can get on IRC and erlang_questions is quite amazing, with most of the discussions tending to fall well on the good side of snarkiness 🙂 As the number of users grows, I am sure that the standard that has been set will be maintained.

Paolo – You were among the Erlang developers that took part to the spawnfest 2012. Can you tell us something more about this two days experience and about the project you implemented with your team mates?

Mahesh – I will be giving a talk about our experience atErlang Factory Lite Vancouver on July 28th. I urge all those who are interested to show up there – there is still time to go, and if you’ve never been to one before, it is a truly memorably experience!

I won’t spoil the talk, but I’ll just leave you with these few tidbits.

We called ourselves Team Net_Split, since the three of us were in three different countries – Argentina, the U.S., and Curacao! I myself was on a Scuba vacation in Curacao, and was coding between dives 🙂

We used the opportunity to start work on our new Project –Erlymob. It is an app that can be used to Crowd-source the sourcing of Crowds, i.e., allow Twitter users to create real-time flash-mobs, get to local-deals with little notice, promote activism etc.

We created just the bare-bones of Erlymob (github here), but as a pleasant and unexpected side-note, also ended up withapp-cache, which is an (evolving) tool for automatically setting up tables, caches, and sequences in mnesia.

There is a whole lot more – you’ll have to wait for the Vancouver presentation to find out the rest 🙂

Some questions, some answers

Paolo – Welcome Pavlo! First of all, would you like to tell us something about yourself and your IT background?

Pavlo – Hi Paolo, thank you for interviewing me. I’m an IT guy with now more than 20 years in the industry. I had an own company, was enterprise architect of some well-known companies and was also leading their software development. But anyway, I’m coder, and will always stay it. Right now, I’m lead architect with codecentric AG, a german consulting company, working on bleeding-edge tech.

Paolo –Why did you started using Erlang? How long did it take to you to handle such a language?

Pavlo – I had to parallelize an interest calculator back in 2005. It was written in C/C++ and had no idea about multithreading. I could parallelize the calculation, so I distributed the calculation using Erlang’s native functionality over several instances and machines. The ops didn’t want it, but since then I’m in love with Erlang. To learn the language – with Prolog experience – took about 2 weeks. But I still learn programming using the libs and frameworks – the ecosystem is huge!

Paolo – How much of your current and previous works is Erlang related? What advantages do you get by using it?

Pavlo – Right now, I’m pretty much in touch in Erlang through Riak. I’m coding hooks and different apps around this wonderful K/V store. Riak is best driven by Erlang apps, so here you go. But I use it as backend for mobile apps, which makes the whole thing much more wider.

Paolo – You are considered a famous conference speaker in Germany and in many other countries: how did you build such a reputation?

Pavlo – That’s new to me. I’m just trying to speak about my experiences, and I’m always tense if people like it. I always feel like having to jump aroung avoiding beer bottles and bad eggs flying into my face. Anyway, I learn speaking and hope people like my work. But still, it’s much easier to write.

Paolo – In the IT field, you are quite famous for your Erlang talks to Java developers: what is the method you use to “convert” them?

Pavlo – I put my thumb directly into the wound. Where Java doesn’t fit the typical use cases, I show how Erlang does. Java is eloquent, everybody knows this. So I also whow the lingual advanteges of Erlang. And as platform for special use cases. But I also show where Erlang doesn’t fit in, and I think that’s the reason why empty beer bottles don’t fly.

Pavlo – Writing is one of the best way to learn. And there is no german literature on this topic, and there are many devs in Germany who would prefer reading in German. So I combined all these things, added some breadth overview I could’t find in other literature, and here I go.

Paolo – Nowadays we really don’t have many books about Erlang, so why did you write your own in German and not in English?

Pavlo – I will not compete with Joe’s, Francesco’s and Richard’s books. I just value them over all. Point. Plus: wait this year, and you will see 5 more books. Awesome times for Erlang!

Paolo – What is the target audience for your book? What can an Erlang newbie learn from it and what can learn an experienced Erlang developer?

Pavlo – I target for experienced developers, no noobs in programming. But you don’t need to know Erlang to read and value it. I’m writing a lot about the VM, the integration with the rest of the world, tools etc. – something, that always is interesting for a seasoned Erlang developer. I always write about professional Erlang development, which is interesting for anybody working in an agile way and using Erlang.

Paolo – Do you provide online resources related to your book? How can users send you feedbacks/errata or questions?

Pavlo –github.com/pavlobaron. You will find the whole source code from the book there. As soon as I get first feedback on errors in the book, I will open a new open repo for errata. Anybody can fork, do pull request etc. That simple.