Thoughts about software development

Erlang

This book gives a pretty good overview of the language and the environment that surrounds it, and overall, I found it fairly enjoyable. I did get lost after a little while, though, but before I explain why, here is a quick review.

Diving in

My curiosity was piqued right from the start, where after explaining to you that the following statement assigns a value to a variable:

X=12.

This variable can then no longer be assigned a different value. In effect, all variables in Erlang are final. Now, this is intriguing and it certainly urged me to read further in order to understand

How it was actually possible to write programs without variables.

Why the language was designed this way.

I won’t spoil the surprise and the book does a very good job at demystifying this aspect of Erlang, but for those of you curious about the answer, I would summarize it as follows: Erlang uses an interesting mix of Unification (also called “pattern matching” and used in Prolog, although Erlang doesn’t have a “cut” operator) and of functional programming, as found in Lisp, Scheme and Haskell.

These names might be intimidating, especially when used in the same sentence, but I mean this as a compliment: Erlang manages to leverage these arcane concepts in ways that are actually fairly consistent and easy to follow. From that respect, Erlang might be a good language for beginners who want to learn more about unification and functional programming.

So far so good

The next chapter describes “sequential programming”, which is just a fancy name to delve into other aspects of Erlang that are still on fairly familiar grounds (at least from a functional programming standpoint). This chapter explains functions, list, list comprehensions (“fancy iterators”), lambda functions and records (Erlang’s version of structures). I wish this chapter had spent a little more time showing more examples of how these various functionalities interact with each other.

I found the next two chapters a bit ouf of place, both in terms of content (an entire chapter is dedicated to exceptions, which is actually an area where Erlang doesn’t differ that much from Java and other similar languages) but also on the examples. Chapter 5, called “Advanced Sequential Programming”, spends a lot of time showing how to manipulate bits and how to write a program that parses a binary image file format. I found these choices a bit odd, especially for a language that works at such a high level. I certainly wish the author had spent more time illustrating some of the other built-in functions shown in this chapter since it would most likely have helped the reader to get ready for what’s coming next…

Chapter 7, “Concurrency”, is the one I was eager to reach.

A new world of threads

If you’ve heard of Erlang these past months or years, it’s most likely because it has the reputation to make concurrent programming easier and faster than any other language so far. This is quite a provocative statement, and I was very curious to see how the language would live up to this claim.

What I find interesting in Erlang is that its functional aspect is a means to an end. The idea is that by using standard functional constructs, such as strict evaluation and single assignment, there is no mutable state in Erlang programs, which makes concurrency much easier to implement: imagine being able to spawn thousands of threads without ever having to resort to locking… This is exactly the premise of Erlang.

Avoiding mutable state is one thing, but you also need to have a way to communicate to other processes. This is typically achieved by sharing variables and carefully locking around them so you can’t thrash their value, but in Erlang, not only can processes not affect each other, they actually don’t have any visibility on their peers, not even their read-only state. When you want to transmit an information to a process, all you know is the id of this process and the data you want to send.

Erlang’s syntax to send messages is fairly simple:

Pid ! a_message

Pid is the process, a_message is any valid value (atom, variable, list, etc…). You receive a message with a structure that basically looks like a switch/case:

receive % receive message sent to this process
% This is a tuple of a type atom and some data
{data, Data_content} -> do_something_else();
end.

Notice that Erlang’s pattern matching fits quite neatly with this paradigm, although I can imagine that such receive blocks can become quite huge as the program grows (I’ll come back to this problem later).

Erlang includes advanced functionalities to manage the most complex architectures made of hundreds of Erlang processes, among which: handling timeouts, spawning and finding processes, distributing them, notifying one or more processes when one of them dies, etc… Erlang also makes it trivial to send code and functions to remote processes, which opens up a lot of possibilities regarding hot-swapping or live upgrading.

The chapters that follow spend more time explaining how distributed programming works, how errors can be handled in massively distributed programs and tries to tackle the difficult problem of fault-tolerant code. They are followed by a few other utilitarian chapters covering the implementation of an irc client, linking to C code, the file API, sockets, ETS and DETS (Erlang databases, which are slightly different from SQL databases, as you can expect).

And then, we tackle the hardest part of this book: OTP and the generic server.

Confusion sets in

OTP stands for Open Telecom Platform, and it’s a set of libraries that were initially written by Ericsson (the author of the language) for their own telecom platform but which have now evolved to make it easy to build large-scale, fault-tolerant and distributed applications. This chapter starts by describing a “generic server” which will become the foundation of all the following examples.

It’s hard to explain exactly what gen_server does, but you can see it as a skeleton including the base logic to send and receive Erlang messages. This code serves actually a much greater purpose and the chapter eventually culminates by showing how a slight variant of gen_server makes it trivial to implement a server that can receive hot updates of the code that it’s running. This is quite impressive, to be honest, but I have to confess my skepticism on whether anyone would want to distribute hundreds of swappable servers on that many machines and then try to manage the upgrades in a way that makes it easy to trace and debug problems…

It’s approximately at this point that the book lost me. First of all, this chapter (and the ones that follow) contain some hardcore Erlang code, so it’s quite possible that it takes several reads (and probably more practical Erlang experience than I have) to understand the concepts better. And to be fair, the author warns us of that much at the very beginning of this chapter.

But what really got my head hurting was the vast amount of repetition that I found in this code. Every listing is basically a clone of the original gen_server code with just a few lines added, removed or altered. This made it very difficult for me to understand what was going on and, worse, to understand the philosophy behind these changes. On the contrary, I found myself having to parse the entire listing over and over again and detect the subtle changes that had been made. And as I progressed through the chapter, I had a clear sense that I was drifting further and further away from the big picture.

This, in turn, led me to wonder what was going on, and I started to realize that Erlang was lacking a fundamental concept that might explain why this code looked so hard to read: Erlang doesn’t have the concept of classes.

Scaling a runtime without scaling a code base

As far as I can tell from the book (please feel free to correct me if I’m wrong), Erlang has functions and modules, which contain a set of functions (each module can import other modules and needs to list all the functions it exports). But there is nothing in-between. I looked hard through the index of the book andon the web, but I wasn’t able to find anything that resembled to classes.

Let me clarify: the problem here is not the absence of classes but of what classes enable, among which polymorphism, inheritance and in more general terms, reusability. As I was going through these repeated listings, I found myself trying to find various ways to refactor them so that only the important part would be shown in the next listing, instead of the entire gen_server. But I couldn’t, and I don’t believe that it’s actually possible in Erlang.

This is a pretty big deal.

Fundamentally, these listings vary in their receive clause and in the various methods they implement. Refactoring a receive clause is like refactoring a switch/case: in general, not very convenient and even if you decide to replace it completely, you need to introduce polymorphism, which Erlang doesn’t seem to support at all. Similarly, it doesn’t seem possible to override an existing method and specialize it, nor is it possible to add a method to an existing module (or have a module extend an existing one). The closest you can come to with Erlang is by using multiple import statements, which is barely safer than #include.

In a nutshell: Erlang goes to great lengths to make it easy for you to distribute hundreds of pieces of code throughout an entire network, yet makes it nearly impossible to arrange this code in a meaningful way.

Looking for trouble

Another dark side of Erlang is error reporting. Here is what happens when you try to assign a variable two different values (probably the simplest mistake you could make):

This is pretty appalling. How can a beginner even remotely realize the mistake they just made? And trust me, it only gets worse from here.

Admittedly, once you start understanding how unification works, you painfully start to learn how to parse these error messages, because they usually all boil down to a failure to unify a variable with a pattern. Still, the whole exercise is very confusing and requires a lot more effort than it should. After eight years in existence, how come nobody has stepped up and introduced a friendly layer translating some of these error messages into something any human can understand, such as “Variable X already has value 1″?

Miscellaneous complaints

The main problem I have with Erlang at the moment is that despite the fact that it was created in 1998 (therefore, after Java, Ruby and Python), the language and its environment feel old. Very old. Most of the examples and listings are run from Erlang’s shell and it doesn’t seem that there is even a decent editor to help you. Besides the interesting functional/unification approach, the structure of the language is barely above that of C with hardly any means to organize or package your code in a meaningful way. For all I can see, an Erlang program is just a set of methods thrown in different modules, with barely any other interaction besides importing available to them.

The section about debugging in the book is merely a few pages long, and besides saying you can use “println debugging” or how to start the Erlang debugger (which seems pretty weak compared to what we’re used to), there is not much to be found there. Which I find very disappointing, because it seems to me that a language that encourages massive parallelism must be ripe for tough debugging problems. Unfortunately, the user will have to take their rite of passage themselves.

Another startling omssion of the book is testing. There is barely a word on how you can test an Erlang program, much less whether there exists a standard Erlang testing framework and, more interestingly, what kind of testing design patterns Erlang supports or requires.

Unfortunately for Erlang, functional programming and unification are a great fit for only a very tiny majority of problems. The fact that C, C++, Java, C# and Visual Basic are the dominant programming languages of our time is not a coincidence: their versatility and the imperative / OO philosophy have turned out to be extraordinarily adaptable and easy to learn and to extend.

No matter how much I admire Erlang’s ability to distribute processes across machines, the lesson we learned in the past decade in software engineering is that it’s easier to distribute data than code, and Erlang’s obsession at sending tiny pieces of beautiful and self-updatable code over the wire therefore appears to be completely misguided. It’s already hard enough to achieve this kind of distribution across a LAN, it’s simply impossible to achieve in any meaningful way on today’s Internet.

Conclusion

I enjoyed the book a lot, and I highly recommend it. It’s definitely at the same level of quality usually found on the Pragmatic Programmer’s bookshelf, and Joe Armstrong is doing a decent job at untangling and explaining a language that’s quite arcane for anyone not familiar with its underlying concepts. I’ll reserve my judgment on some of the sections where I got lost and I’ll do my best to try and reread them again in the near future, but I already know that the extensive repetitions in the listings make the endeavor difficult to tackle.

The main problem is Erlang itself, which is very interesting from an intellectual standpoint, but which I don’t see displacing any of the mainstream languages in a very long time…

This entry was posted on September 18, 2007, 3:26 pm and is filed under General. You can follow any responses to this entry through RSS 2.0.
Both comments and pings are currently closed.

21 Comments

Some thoughts for you:
– Editor: Emacs. A bit of an adjustment if you’re used to lots of auto-everything, but it’s a good thing to think carefully about your libraries and imports.
– Distribution granularity: Coarse. Small messages, big computations.
– Classes/Methods vs. Functions/Values: A class without the methods is state, and Erlang doesn’t have that other than the currently enabled receive (or buried within something like ETS/DETS). Erlang does support polymorphism in records. (See, e.g., the is_record bif.)
– Higher-Order Functions: Higher-order functions and some of the sugar that goes with them are a huge hammer. Operations that would require quite a bit of Java code collapse into a few dense expressions, although that’s a mixed blessing.
You should dip a toe in Haskell and OCaml now, and you’ll get a more complete sense for FP.

Message based concurrency provides significant advantages over typical locking strategies, but Erlang is just too large a leap from today’s popular languages. Concurrency is just one problem and as you’ve noted there are noticeable gaps in the Erlang toolset.
To help bridge the gap, I created Retlang.http://code.google.com/p/retlang/
It provides powerful message based concurrency while leveraging the strengths of .NET. Elegant concurrency requires the Erlang mindset, but it doesn’t require a new language.

> Let me clarify: the problem here is not the absence of classes but of what classes enable, among which polymorphism, inheritance and in more general terms, reusability.
Inheritance is kind-of enabled by using behaviors (which is what gen_server is), based on something similar to the GoF’s Template Method.
Reusability is performed on two levels: the module level (which only works for some things, but is good enough for, basically, “library” code) and the OTP Applications level. If you want to reuse heavy code, you extract it into an application and use it in a kind of Service-Oriented architecture: each OTP application is a service that can be used by any part of your software.
Creating OTP applications yourself is not the easiest thing there is, but sticking to OTP guidelines/recos/whatever (once you’ve understood them, which is … not easy) allows you to do it.
> Refactoring a receive clause is like refactoring a switch/case: in general, not very convenient and even if you decide to replace it completely, you need to introduce polymorphism
Replace a receive? By what? If you have receives of more than half a dozen cases you’re usually doing something wrong.
> it was created in 1998
It wasn’t, it was *open sourced* in 1998, the first implementation was created in *1987*, 5 years before Python and 7 before java.
> an Erlang program is just a set of methods
Erlang uses functions, not methods, if you think in terms of methods and POO of course you’re going to hurt and fail.
*Erlang doesn’t use object-oriented programming, don’t think in terms of objects and methods, think in term of processes and functions*.
> Erlang’s obsession at sending tiny pieces of beautiful and self-updatable code over the wire therefore appears to be completely misguided
I don’t get that one, yes you can send “pieces of […] code over the wire” (I assume you’re talking about sending *funs*), but I fail to see how it’s an obsession of erlang, functions are data and erlang can send any data over the wire, that’s all there is to it. Nothing *forces* you to send code over the wire, in fact you usually don’t, you spawn processes with static code and you just feed them data.
And the gen_server example shows that you can send new code to your server, but it’s just that, and example.
As far as code swapping/reloading goes, erlang coders will usually use the built-in hotswapping (replace the BEAM file, then send a message telling the processes to reload themselves, or just kill the task processes and let the supervisors respawn them with the new code, or have the processes regularly go through a code-reloading task, or whatever).
Finally, don’t mind not quite understanding OTP in general or gen_server in particular too much: they’re quite complex and deep structures (and the documentation itself isn’t anywhere near the quality of the language, neither in interface nor in content, surprised you didn’t mention it by the way).

Sorry, but you’re wrong on many account.
As soon as you want to write a system that
interacts with any sort of events you would
benefit from using Erlang. What people seem
to not realize is how ‘easy’ it is to write a
24×7 system in Erlang that handles massive load
and can upgrade its SW without disturbing the
traffic. This is the true power with Erlang.

The ‘retlang’ above mostly sucks; it’s not performant, and certainly doesn’t give the same level of Erlang as Erlang does. Erlang’s actors work because there’s no shared state, which Retlang certainly does have; it’s a thin wrapper about creating a bunch of threads (one per object).
You should look at Scala, both from the language perspective, but also at the actors package in particular. This suffers from the same issues as retlang (you can have shared state which causes problems) but it’s much more performant, since the threads are dissociated from the unit of work. And, since it’s hosted on the JVM, you can also use it to call Java code, whilst retaining the elegance of functional languages like Haskell.
Oh, and did I mention it’s hybrid functional/OO, and supports pattern matching (albeit to a slightly lower level of power than Erlang)? So you can have your class and eat it.

>>The ‘retlang’ above mostly sucks; it’s not performant
Retlang can handle hundreds of thousands of messages per second. How performant do you want it to be?
>> and certainly doesn’t give the same level of Erlang as Erlang does.
Absolutely true. It is written in .NET after all.
>> Erlang’s actors work because there’s no shared >> state, which Retlang certainly does have; it’s >> a thin wrapper about creating a bunch of >threads (one per object).
While you can certainly create a thread per object in Retlang, that is a design choice and never required by the framework. An entire message based application can be written utilizing a single thread and thousands of objects if needed. The choice is yours. As for shared state, it is not always a bad thing. It’s just another design choice.

I was also intrigued by Erlang couple of weeks ago when I read some comments on its parallelism paradigm and did a bit of reading (but as much as you).
I am not sure how synchronized access is never done. For a simple usecase say: to keep count of the number of users accessing a system – one would need to have a shared userCount that is updated from all system entry points. I assume a single process would be dedicated to do this update and provide some kind of synchronized access. Even if it is stored in a database it qualifies as synchronized access. Not sure I see how this can be done otherwise.
I wonder how the concept can be adapted to Java. I see some similarities with the Servlet / HTTPSession model. Encapsulate the HTTPSession with a regular expression API for read & writes of all its key-value pairs (simple types only). The API would issue copies for key-value reads and synchronize access to key-value writes in memory and/or a database. This design could be made very scalable if the HTTPSession is persisted/distrubuted.

I agree that Erlang is bemusing in a number of ways. I blogged about this today:http://blog.robjsoftware.org/2007/09/messages-and-transactions-part-1-erlang.html
Your post inspired me to accelerate this one a bit, to make the discussion more timely.
I agree that object-oriented code structuring seems absent from Erlang. More puzzling to me, though, is the lack of serious detail about how online code upgrade and/or extremely high reliability are actually done in Erlang systems. I was disappointed by the lack of implementation detail in Armstrong’s thesis.
I am sure there is a lot to learn from the actual history of high-availability systems development in Erlang… but right now I’m not sure where it can be learned! None of the Erlang references I’ve yet seen get into the nitty gritty.
Cheers!
Rob

Matthew, if you wanted to track a user count you’d have a process that tracked it and send it messages when users logged on or off. You could have it return {error, too_many_users} in the reply to logon when your usage count is exceeded, instead of a routine reply (usually ‘ok’).

Nice review. After spending some time trying to understand the underlying mechanisms I realized that there was a big paradigm shift, especially from imperative programming. It was important for me to unlearn some things, before I could appreciate the differences.
Secondly, I think priorities and purpose of Erlang, as a language, are different. Most of the features and lack of some, can be attributed to that.

On easier concurrency due to immutable states… It’s been long known in the OOA/OOD/OOP circles that OO over immutable objects is 100% doable and it does solve a lot of problems. I’m in that camp and my OOP in Java creates a *lot* of immutables objects (Java is an ugly wart to me for it allows oneself to shot in the foot so easily, but it’s possible to approach some kind of relatively ok OOP once its ugly warts are known). Btw, if you think of it for half a second, the circle/ellipsis problem lies in the mutable setXXX(…) method, not in a fundamental OO problem. One day, one day some clueful people will see the light… But I’m not holding my breath and I won’t come from people who can’t think outside the 3GL level (as soon as you start thinking outside the 3GL level you realize that a lot of problem are actually non-existent and not worth worrying about). Respect to Erlang.

Hi Cedric,
I was reading the book too – and after our last conversation I wanted to wait for your review. Most – if not all – of your complaints surfaced during my review of Erlang too.
The main point about Erlang is OTP and how much better Erlang scales than threads with Java.
But when reading about Scala, which implements the same receive concurrency model as Erlang, the question comes up: Is scalability an Erlang thing? Can Scala with actors on the VM scale as good as Erlang?
(And the only remaining feature question about Erlang is: can a future VM provide the same hot code exchange as Erlang does? I guess if this is important to people, the VM will support it.)
I later blog my thoughts about why Erlang will not replace any other language.
—
Stephan Schmidt :: stephan@reposita.org
Reposita Open Source – Monitor your software developmenthttp://www.reposita.org
Blog at http://stephan.reposita.org – No signal. No noise.

dbt,
I think we said the same thing.
I said “I assume a single process would be dedicated to do this update and provide some kind of synchronized access”.
U said “if you wanted to track a user count you’d have a process that tracked it and send it messages when users logged on or off”.
I presume all messages will queue up to the process that tracks the usercount – in which case this would be synchronization.

I think it’s great that more people look into the strengths of Erlang, and that some are inspired to develop similar capabilities in their own favourite language.
As for scalability and reuse, there is actually quite a lot of evidence that Erlang gives a higher degree of reuse and much fewer lines of code than in comparable C++ programs. But it’s really a shift of mindset compared to OO; if often takes a few months to shed the OO thinking and approach Erlang, Haskell et al for what they are. In my experience, those who do, will not look back.
Regards,
Ulf Wiger

Didn’t read the book, but am pretty well versed in functional programming. Your quote:
“Erlang uses an interesting mix of Unification (also called “pattern matching””
is misleading. Unification != “pattern matching”.
Unification is used in the determination of the most general unifier (MGU) in functional programming languages, for typing purposes. It recursively attempts to determine the “largest type” of the sub-expressions inside larger expressions, with typing rules determined by the language semantics.
Take a compilers course at a top CS program, and you will have your fill of unification.

I make some comments about this on my blog – perhaps you’ll find them interesting?
Your validation is preventing me from pasting a link, so you’ll have to decode:
tim-watson dot blog spot dot com
ps: (blog spot) without spaces.

As someone wrote earlier, don’t try and map standard OO concepts like classes and inheritance on to Erlang, trying to do this end up just confusing the issue and you and not get you anywhere. Erlang is a functional language, not an OO one, and the basic code structuring primitives are modules and functions. It uses concurrency, processes and message passing to structure an application.
How can you say that Erlang doesn’t support polymorphism? I think you have missed the point. You define the interface to a process, its API if you will, through the messages you implement. How you interpret the messages is in the code for the process. Different code, different handling. This allows you to have many types of processes with the same API so you can build any type of polymorphic system you want. The difference is just that you don’t use classes to do it. Polymorphism is not unique property of OO, quite the opposite.
If you must try and view Erlang with OO coloured glasses then you should base it on the view of Alan Kay who wrote “OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things.” In this sense Erlang is very OO, even more than say Java, and even Joe admits it.
One final point is about the Erlang model of concurrency. It is usually interpreted as lightweight processes and asynchronous message passing. While these are fundamental properties of it you must also include the error detection and handling mechanisms which are integrated into the concurrency model. Without these you do *NOT* have Erlang style concurrency, only a rather weak copy.