After reading http://clojure.org/rationale and other performance comparisons between Clojure and many languages, I started to think that apart from ease of use, I shouldn't be coding in Python anymore, but in Clojure instead. Actually, I began to feel irresponsisble for not learning clojure seeing it's benefits.

Does it make sense? Can't I make really efficient use of all cores using a more imperative language like Python, than a lisp dialect or other functional language? It seems that all the benefits of it come from using immutable data, can't I do just that in Python and have all the benefits?

I once started to learn some Common Lisp, read and done almost all exercises from a book I borrowed from my university library (I found it to be pretty good, despite it's low popularity on Amazon). But, after a while, I got myself struggling too much to do some simple things. I think there's somethings that are more imperative in their nature, that makes it difficult to model those thins in a functional way, I guess.

The thing is, is Python as powerful as Clojure for building applications that takes advantages of this new multi core future?

Note that I don't think that using semaphores, lock mechanisms or other similar concurrency mechanism are good alternatives to Clojure 'automatic' parallelization.

4 Answers
4

Can't I make really efficient use of all cores using a more imperative language like Python, than a lisp dialect or other functional language?

You definitely can. Depending on the type of the problem (e.g. process clearly separable portions of some big computative task in parallel), it can even be fairly easy. I guess most of the concurrency in the world is still done directly in imperative languages, although the paradigm is shifting towards functional solutions.

Python isn't exactly known for its concurrency capabilities, though. An example of a well-established imperative language with excellent concurrency support is Java.

It seems that all the benefits of it come from using immutable data, can't I do just that in Python and have all the benefits?

The trick with immutable data is that, if you do it with the traditional, simple data structures such as arrays, you'll end up with massive copying and garbage collection, ruining performance. Instead, you want effectively immutable data, which can be implemented with various persistent data structures, as it's done in Clojure under the hood. Such structures could be done in pretty much any language as libraries, but of course direct language support is always nicer.

Python is extremely bad for applications that need multiple threads, but it's caused by deficiency in the available virtual machines, not the language itself. Python's generators would really be good candidates for implicit parallelisation. The problem is that the standard implementation, CPython, can only execute python code on one CPU, only calls in native libraries can run in parallel. The PyPy runtime plans to fix it, but is not there yet and while the Jython and IronPython implementations don't have the limitation, they are rather incomplete (they don't have much of the standard library, you have to use the standard library of the hosting environment in them).

However there are many other languages designed and implemented with parallelism. Of course parallelism is much easier in functional languages, which is why all the languages with good parallelism support are either functional, or have strong support for functional programming. I can think of:

Haskell, the most used pure, non-strict functional language. It's pure non-strict nature allows the compiler to parallelise some things implicitly.

Erlang is a strict functional language designed specifically for implementing robust parallel servers.

Clojure, of course. Like other lisps it is designed for functional programming, but supports imperative programming as well.

Go is a new compiled procedural language designed for concurrency, though neither the runtimes nor the language itself are very mature yet.

@Sonia: ... which is what I call not very mature. It usually takes 10 or more years for language specification to settle. I at least expect Go to follow Java and C# and get generics one day.
–
Jan HudecApr 11 '12 at 8:25

Note that I don't think that using semaphores, lock mechanisms or other similar concurrency mechanism are good alternatives to Clojure 'automatic' parallelization.

Actors are an object-oriented way of abstracting from the underlying threads / processes: you basically only see objects running concurrently and sending messages to each other.
So, if you want an object-oriented way of implementing parallelism I would
also take a look at languages (or frameworks) supporting the actor model.

Concurrent programming is usually hard because of "sharing state". Functional programming may not be the ultimate answer but it surely make it doable without losing your hair.

Clojure for sure is a viable option but even with its excellent concurrency tools, you might need to build something else. Check Prismatic for an example:

Our backend language choice is Clojure on the JVM. There are other great functional languages on the JVM, like Scala, but what we like about Clojure is that it has a small, simple core focused around processing data represented in its excellent set of persistent data structures (or their mutable java.util equivalents, when the need strikes). While we make heavy use of the core of Clojure, we don't use its concurrency primitives (atoms, refs, STM, etc.) because a function like pmap doesn't have enough fine grained control for our needs. We opt instead to build our own concurrency abstractions in Clojure on top of the outstanding java.util.concurrent package.

Julia does not impose any particular style of parallelism on the user. Instead, it provides a number of key building blocks for distributed computation, making it flexible enough to support a number of styles of parallelism, and allowing users to add more.