Friday, 10 July 2009

a groovy scala example

So my last post caused a bit of a stir (though some excellent comments!). Sorry about that! :) I think a few folks got the wrong end of the stick thinking my post was some kinda put down of dynamically typed languages. I still like Groovy/Ruby/JavaScript etc. I tried to focus the blog post purely on folks who were hacking statically typed code in javac; that Scala was a natural long term replacement which is a much better, powerful & more elegant statically typed language. If you've already stopped using javac and using Groovy/Ruby/JavaScript/Clojure/whatever then the post wasn't really meant for you as you don't need a javac replacement as you've already stopped using javac :)

Scala, it must be stated, is the current heir apparent to the Java throne. No other language on the JVM seems as capable of being a "replacement for Java" as Scala, and the momentum behind Scala is now unquestionable

Anyway the reason for this post was to show how Scala is very Groovy based a nice example posted to the comments by James Iry. Here's the Scala code first

val (minors, adults) = people partition (_.age < 18 )

I'm sure most of you will agree its pretty clean, concise code and easy to read. It partitions a collection of people into two collections of minor and adults based on age.

Notice how apart from some pretty minor surface syntax differences they are kinda identical to read. Mostly Groovy forces the dot between people and the split method (I think thats still true?) and uses 'it' for the default parameter name in a closure block rather than the Scala equivalent of _ if no name is specified. (Scala the _ is the wildcard symbol in various parts of the language as * is an operator on lots of objects like numbers).

So Scala and Groovy code can look about the same; nice concise, easy to read - describing the intention of the developer, rather than lots of noisy lines of code, or maybe ugly anonymous inner class noise using non-standard helper libraries to try simulate having nice core language support for working with very common data structures in a powerful expressive way.

There's a ton of Java libraries out there trying to tack on the ability to work with common collections & data structures in the modern Scala/Groovy/Ruby/Python/C#/VB way (dare I use the monad word here?) - I even helped create one many years ago. FWIW if you're stuck on Javac, do take a look at google collections which is really good.

My main criticism of these approaches in Java is they are non-standard (there's a ton of them out there - which to use?) and they are separate to the objects you want to work with. You can't just use a List, Set, Array or Map directly - you have to remember the separate class in a separate package you need to import so you can just do simple basic operations on data structures concisely. This kinda stuff is so basic, it should be a core feature of the programming language you use IMHO.

So both the Scala and Groovy examples above are a big improvement IMHO over their Java equivalents. I just wanted to demonstrate one of the additional benefits the Scala version has. While the code looks about the same, the Scala one creates immutable variables (which are great for concurrency) but the main difference is that the values are all statically typed. That means the compiler / IDE / documentation tool knows the exact types of the variables (minors, adults, people). Big deal I hear some of you say.

Well here's a handy example of why that is useful. Imagine you exposed this line of code as an API folks could invoke. Here's a PersonThingy object which defines a method minorsAndAdults as follows

def minorsAndAdults() = people partition (_.age < 18)

Now notice how the generated API documentation describes the return type of minorsAndAdults() as being (List(Person),List(Person)). i.e. you know its gonna return a 2 value tuple with a list of people for both values. This is very handy when you want to expose your code to other people; you don't have to write documentation (which gets out of date fast) describing what it is you return, the compiler can do all this for you. Plus when folks try to invoke your method their compiler & IDE will type check their code as they type it to ensure its used correctly (e.g. in case the user forgot to extract the first value from the tuple etc).

This also works nicely when using map. For example here is a little example which defines two methods greetings and lengths using the same value and a map method call but just using different expressions in the function passed

def greetings = names map ("Hello " + _)

def lengths = names map (_.size)

which return List[String] and List[Int] respectively.

So while dynamically typed languages and Scala can look as equally as concise; being statically typed can save you yet more typing explaining stuff the compiler/IDE/documentation generator already knows - and it avoids documentation going stale after refactoring - for example to change the data structure.

One closing observation. A common put down (usually from folks who've never read a good Scala book or really tried using Scala in earnest) is "oh its too complex". Yet the code you frequently write when working with objects, collections and data structures often looks very similar to the Ruby/Groovy/JavaScript/Python equivalents - yet folks rarely use the complexity word with those languages.

I wonder if part of the problem is, Scala & its community tends to describe Scala code and what the language is in terms of various (slightly academic) programming language concepts so that you can understand how the language works internally so advanced users can see how you can bend it to do what you need in a more concise & expressive way. For example back to James Iry's example

val (minors, adults) = people partition (_.age < 18)

That one example uses a tuple, pattern matching, a lambda, and partial function application. Describing how the language works tends to make it sound more complex (especially if you use the monad word) than just saying 'here's how to partion a collection'. If you were to take the similar Groovy/Ruby/Python code and tried to explain how the language actually implemented that line of code it would sound about as complex (often with meta object programming in there somewhere).

Scala is frequently described as a unification of functional programming and OO programming language concepts. A common push back from OO folks is, "I don't like functional programming, I just wanna do OO". Yet those same OO folks seem to really like closures, lambdas, for/list comprehensions and so forth. For example VB has lambdas, C# them too and has list comprehensions. Folks tend to really like them - whether they are developers using VB, C#, Ruby, Python, Groovy, JavaScript etc.

i.e. it seems most of us all really like an OO language with functional language features; we just don't like to think of ourselves as functional language developers :) Microsoft BTW is doing really well at turning C# and VB into OO and functional languages - but just avoiding using the 'f' word to avoid scaring folks off. I wonder if the Scala community should use the 'f' word a little less? :)

31 comments:

Maybe it should indeed. I especially don't like the way the new ScalaCareers.com site seems to equate Scala with functional programming.In my use of Scala, I use closures, lambdas, pattern matching where they are useful, but I wouldn't say what I'm doing is functional programming.

This example is interesting, and it shows both languages are pretty close syntax wise. But the Groovy example is a bit more polymorphic in the sense you can partition anything that has some age property. In Scala, you would have to introduce some traits, or use some anonymous types. So the equivalent code to be as flexible as the Groovy one would be way more verbose. So both examples are not really equivalent,

With Javascript I have had some of the most verbose programming experience ever and just my Ruby mode for syntax highlighting and real-time editing for my Javascript + Canvas based Text Editor (which also has support for Javascript, Python, and a little support for Ruby Tenjin, HTML and CSS), already runs over 3500 lines of code (counting spaces).

Javascript is plenty verbose and coming from Ruby I find some of that verbosity kind of depressing at times.

What Javascript buys me is a very repetitive programming experience more in-line with what we used to expect in the "older" programming languages. For example, with Javascript I am back to writing for loops almost all the time even if I could try to use other constructs to avoid some of it. The reason I use for loops is also to avoid the overhead of calling methods too many times in tight loops.

Also, Javascript is mainstream and has had some good implementation improvements as of late. Javascript performance in the V8 engine by Google can often beat most of the popular scripting languages out there and the Google V8 team is expecting to keep on doubling its performance for the coming years (Moore's law? LOL.)

I am currently puzzled by having such dissimilar scripting/dynamic programming languages in Ruby and Javascript competing so well for the programmers' heart.

The Scala code is polymorphic too -you can use the exact same line of code on any collection of objects with an age property (method/field). Its just the Scala one won't compile if your collection of objects don't support the age property/field/method - whereas you get a runtime exception in Groovy.

As @Michael says, if you have no idea what objects are in the collection at compile time, you could use structural typing to get the duck typing of dynamic languages if thats what you really want - but I doubt most folks really want that; when they write the code they know what they are iterating over so they want the compiler/IDE to smart complete on what methods are available on the collection element & give a compiler error if they make a typo etc.

Structural type inference would definitely be the most obvious way to make the partition code polymorphic in a duck-typing fashion. For the record, the ideal would be to infer a type for the `people` list:

val people: List[{ def age: Int }] = ...

people partition { _.age < 18 }

There, that's just as polymorphic as anything Groovy could do. Or is it? Why restrict `age` to just Int, why not *anything* which defines the < method?

This is now an extraordinarily flexible bit of code. However, the Scala compiler cannot handle it out of the box due to the fact that SaneNum is a recursive type (it refers to itself). As any good type theorist knows, recursive types must be treated with some care, particularly in practical compilers.

I have two points in all of this: firstly, while Scala is technically capable of matching Groovy's dynamic duck-typing in this particular circumstance, it comes at a price both in complexity and performance (structural types are implemented using reflection). Because of all of these contortions, I would argue that Groovy is inherently more polymorphic. Given that it is a dynamic language, this isn't really surprising.

My second point is that structural type inference is a very viral thing to do to a language. Intuitively, we only wanted one structural type (for the `age` method), but we almost had to continue until every type involved (except for List) was structural. It's very hard to just draw a line and say that one part of the type system is nominal while the other is structural. In fact, until Scala came along, there were those who thought that the two would *never* mix without compromising soundness. Scala *could* have structural type inference of the form demonstrated here -- and languages like OCaml actually do -- but this sort of inference is very much at odds with Java's type system. If Scala went this direction, it would lose a lot of its incredible interoperability.

Daniel Spiewak illustrates the edges of Scala's typing when it comes to dynlang-like "polymorphism" very neatly. Scala can appear to many things a dynlang does, but it can't do all of them. A few others that come to mind are rampant metaprogramming (discussed a bit in James's previous post), "method_missing" handling, and any runtime changes to types, like dynamically mxing in a new module based on runtime feedback. Most of these things fall well outside what you can do with a static compiler, and yet they're what draws people to languages like Ruby.

All that said, I do have a few more intangible gripes about Scala.

First off, every time I've questioned why something is the way it is in the Ruby community, I've gotten a friendly discussion. It may have been a question heard a dozen times, and often people recommended just searching the mailing list archives, but responses are almost always very cordial. When I have questioned the way things are done in Scala, usually the answer is "because that's the right way...go read this paper and come back." The Scala community (or at least the most vocal members) seem to look down their noses at anyone with a doubt about Scala. That's extremely damaging, and needs to change.

Second...I really like some Scala code: Scala code I have personally written. I am a beginner at Scala, certainly, but some of the code out there looks absolutely terrible. There's too much temptation to use too much magic, and the infinite freedom to define symbolic operators absolutely terrifies me. The truth of the matter is that almost anyone can look at a Groovy program or a Ruby program and know what it does at a glance. And yes, that may apply for many Scala programs, especially one-liners like those being bandied about in these blog posts. But there's a *lot* of Scala code out there that's totally baffling to someone that doesn't know Scala. Some of this could be a lack of Scala "best practices", or just new folks experimenting with every bit of sugar they can find. But the ease with which one can write indecipherable Scala code is something the Scala community should really address.

I like Scala, and though it isn't perfect (THAT'S RIGHT, I SAID IT) it currently appears to be the best candidate for taking over a large portion of static-typed programming on the JVM. But much of what dynamic languages like Ruby can do is still not possible in Scala, so I believe we're still going to see Java platform development trending toward a nice mix of static and dynamic languages working together.

You're absolutely right about general reaction from the Scala community. I think it's probably an attribute of well-meaning academics more than genuine malice. Those who hold PhD(s) in programming languages actually find heavily-theoretical papers to be the best explanations for a lot of things, including why Scala is the way it is on a fundamental level. To be honest, I can identify with some of that attitude (even lacking the PhD); Scala's theoretically sound and consistent foundations are on of the things which really drew me to the language, even before I fully understood them.

The point is that the Scala community does need to be more considerate on such matters. Scala has deep reasons for doing a lot of the things it does, but that's no excuse for just brushing off newcomers with utterly unhelpful advice like "go read such-and-such a paper".

As to unintelligible code, I've actually seen more indecipherable code written in Ruby than I have in Scala. To some extent, I think you're right that this is a matter of best-practice. Ruby best practices are generally defined by Rails, which does a good job of effectively controlling all of that meta-programmatic goodness. Scala doesn't have anything like that just yet, though there does appear to be an almost-subliminal development of de facto best practices. Given time, I have complete faith that Scala developer mindset will catch up with the Scala language. Until then, I guess we just keep muddling through!

Scala isn't perfect, and neither is its community. We do our best though! :-)

>Scala can appear to many things a dynlang does, but it can't do all of them.

That's true. What I showed in my Feel of Scala talk on Parleys.com is that I could get several of the same benefits I got from dynamic languages using static techniques in Scala. One I showed was the ability to add (or at least appear to add) a method to an existing class. This is done with open classes in Ruby, and implicit conversions in Scala. But with open classes I can still do two things I can't with implicit conversions. With open classes I can remove a method from an existing class, or change the implementation of an existing method in an existing class (as far as I know). I can't do that in Scala, but I don't mind, because that wasn't what I liked about open classes in the first place. I liked being able to add a method, like the "should" method in RSpec or ScalaTest matchers, and implicit conversions in Scala let me do that.

I showed two ways of doing duck typing in that same talk, one with structural types as some have pointed out in the comments here. But I also used duck typing to test private methods through reflection. Structural typing still gives me deterministic refactoring, and with a compiler plugin, I was able to get deterministic refactoring across my invocation of private methods through reflection.

What I didn't show in that talk was method_missing, which is another thing I like about dynamic languages: the ability to implement a method on the fly when it is invoked. There are ways to get that same benefit in Scala. I didn't show them because I haven't had to use any of them in ScalaTest yet, and I wanted to show only real solutions to real problems in that talk. But one thing you can do is static code generation (imagine generating finder methods from a database schema). Another thing is an internal DSL (Lift has one of these for accessing databases, for example). And the third is to simply implement the dynamic technique in Scala, like I did for private method invocation.

An example of this is actors. An actor message send is basically a dynamic method invocation. If the actor you send the message to recognizes your message, it will handle it. An actor can be written so that it handles unrecognized messages in some way, and that is exactly the same concept as method_missing in Ruby. Actors process messages asynchronously, but there's no reason you couldn't make a dynamic database access layer just like ActiveRecord in Rails that provides dynamic finder methods with syntax like:

user findBy 'cityAndState(city, state)

That said, normally people wouldn't do it that way, because usually the static alternatives (code gen or internal DSL) are better because with those you get deterministic refactoring. But the point is you *can* use the dynamic method_missing technique in Scala if you want.

>When I have questioned the way things are done in Scala, usually the answer is "because that's the right way...go read this paper and come back." The Scala community (or at least the most vocal members) seem to look down their noses at anyone with a doubt about Scala. That's extremely damaging, and needs to change.

That's a bummer to hear. I have seen some of that too, and pushed back against it. My perception has been that initially Scala attracted folks from the functional programming camp, and a very small but vocal few of them seem to have some kind of superiority complex with respect to "the rest of us." But it hasn't seemed to dominate. For the most part people have seemed to get respectful, thoughtful answers to their questions on the list. Where did you ask your questions?

>Second...I really like some Scala code: Scala code I have personally written. I am a beginner at Scala, certainly, but some of the code out there looks absolutely terrible. There's too much temptation to use too much magic, and the infinite freedom to define symbolic operators absolutely terrifies me.

I've seen some pretty scary looking Scala code too, but I have also seen some very readable Scala code, including code not written by me. I think one of the tradeoffs you get with Scala is it gives developers a lot of power to make the code look like whatever they want it to look like. This means it can be made to look extremely readable and clear, but also extremely complicated and nonobvious. I think the only thing we can do about it is promote best practices, but even then there will be the occasional travesty of readability.

I have to agree with Charles. I've looked into Scala some and liked I lot of what I saw. But being a Ruby programmer I thought to myself, how do they live without metaprogramming? Sure metaprogramming can be abused, but it's also the basic for ActiveRecord, which I think is the best ORM I've ever used. Metaprogramming ends up being a great way to create extensible abstractions, when done well.

So I decided to crack open the source code for Lift and then I started seeing stuff like this:

Huh? Maybe this would make more sense in time, but it seems to be borderline ridiculous to me now. Of course the same could probably be said about the internals of Rails and it probably only seems clear to me because I understand Ruby very well at this point. Once I understand Scala more, the above code might make sense, but right now, it's Scary.

I agree that type safety is nice to have, but that has to be balanced against the complexity. After all, one of the advantages of type safety is that it is supposed to make the code easier to understand, but at some point if the type system becomes so complex, it gets in the way more than it helps.

> I agree that type safety is nice to have, but that has to be balanced against the complexity. After all, one of the advantages of type safety is that it is supposed to make the code easier to understand, but at some point if the type system becomes so complex, it gets in the way more than it helps.

I agree. One thing I have found about Scala is that it reduces the cost of static typing quite a bit compared to Java. But it isn't free.

>I think the ScalaTest DSL looks great and is clearly as elegant as the syntax of RSpec in Ruby. But comparing Lift's Mapper with Ruby's DataMapper, Ruby is the clear winner.

Thanks. I would also agree about a Lift Mapper versus Rails ActiveRecord comparison, but I don't have any experience with the DataMapper in Ruby. Nevertheless, I would point out that what we're comparing here is libraries in languages more than the languages themselves. There are many other ways to design database layers and matcher DSLs in either Scala or Ruby, so it is a bit unfair to either languages to make a judgement on readability based on one or two library data points.

@Paul, David Pollack, the creator of Lift, once made the comment that users of Scala libraries don't have to understand the details of the type system as much as library designers, because type inference hides a lot of details. There's a kernel of truth there, but the problem is that users have to read the "scaladocs" (or sources) of the libraries and then they get smacked in the face with the type stuff. It's a dilemma.

Martin Odersky has said that one of his long-term goals is to promote better tooling for understanding types, debugging problems, etc., without having to understand the intricacies of Scala's type system (or syntax).

I'd love to see approaches for API documentation that give you just enough information to use the API without confusing you with stuff that type inference will handle for your.

It's nice to hear the leaders in the Scala community are identifying aspects of the language that give people trouble and are actively thinking about what can be done to improve that. I think that attitude goes a long ways in making a language successful, as Charles alluded to in his previous comment.

I hear you. I think some of the Lift code could be refactored a bit to make it a little easier to understand (it can be quite hairy to understand particularly as a Scala newbie). It is possible to have concise yet relatively easy to understand code in Ruby and Scala IMHO.

Though I remember struggling first time I tried to understand the rails internals too. The scala syntax can be harder to grok (particularly when using traits, generics with contravariance/covariance stuff - though you soon get used to that) - though on the plus side its much easier to navigate due to static typing. I remember wondering where things might be in rails code & doing tons of greps which you don't need to do usually with scala code.

However be that as it may, I just wanted to comment on database mapping stuff in general though. In the last 20 years or so of working with databases, I've found database schemas change rarely - plus once things get into production (when the hard work has only just begun) then only backwards compatible changes tend to take place (optional additions of new columns etc). DBs soon get huge and hard to remember the tables/columns. For me this is a perfect scenario for using static typing so you can use easy code completion, let the compiler/IDE catch your typeos on table/column names & have nice auto-generated documentation for your model code (which you use *alot* but rarely change).

Even when using rails you tend to define your model in the rails migrations DSL; so you are explicitly describing your model anyway (not just relying on whatever the DB structure is at runtime) - so it seems way simpler to just define that model once in a statically typed way you can then get lots of benefit for during the development & support of the application.

That being said, Rails is still much more mature & polished than Lift; but Lift is progressing nicely IMHO. I'm a huge fan of Rails; I think its an amazingly impressive piece of work & its been a total game changer in the web framework space - though I'm now preferring Lift (particularly in tandem with JAXRS when building restful services).

First off, I quite like Scala. That said, it almost immediately made me move to Haskell but more on that later.

I don't believe Scala will become the "next" Java. The reason for this is that Java is very readable and I believe that that readability has contributed significantly to its success. It takes some effort to write obfuscated Java and it is usually immediately obvious that it is obfuscated. The same is not true for Scala. It always amuses me when someone demonstrates some code in Scala only to have innumerable comments suggesting more concise alteratives, all of which use completely different mechanisms to achieve the same end. This does demonstrate Scala's power and flexibility but it also means that the overhead in grokking a piece of code that you haven't written yourself becomes larger. Even in Java, it is trickier to read someone else's code if they don't use the same idioms. In Scala it is particularly tough.

IDE support. This is often not viewed as the advantage that it is. What kind of language is so crippled that it is only useable when you have an IDE? The reality is that Eclipse / Netbeans / Idea do exist and they're quite good. As a result, a lot of the Java limitations are, if not resolved, at least ameliorated. Anonymous inner classes are almost closures and Ctrl+Space is your friend. Most importantly, however, Java does not lose its readability. Type inference in Scala is awesome. The problem is that without knowing the types, and they could be several dependencies away, it really isn't obvious what a piece of code means, especially if someone has gone and overloaded some operators. That's not to say a Scala IDE couldn't resolve this particular issue.

My experience with Scala has been similar to many of those expressed in the comments. The documentation when it comes from the academic side of the world is completely foreign. Which makes learning Scala a bit like learning to cook in a foreign language. It is surprising that folk who appreciate conciseness in a programming language believe PDF based papers are the most efficient way of imparting knowledge.

Where Scala is interesting is in its encouragement of immutable programming and support for concurrent programming through Scala Actors. I do believe that whatever replaces Java needs to do for concurrent programming what Java did for memory management. The problem with Scala, however, is that it needs to support Java and the compromises are just too large. It reminds me of C++ in that respect.

None of these arguments speak to the part of me that enjoys programming, however. That side of me doesn't require an IDE, nor cares about who can read my code tomorrow, including me. The problem with Scala is then that it is too much like Java. The compromises mean that Scala doesn't feel consistent. In striving to bridge both the imperatives and functional worlds, Scala loses some elegance. Which is why for pure entertainment, I skipped straight to Haskell.

I don't believe Scala will become the "next" Java. The reason for this is that Java is very readable and I believe that that readability has contributed significantly to its success

The thing is; each line of Java code may often be easier to read & understand (or it might not since it might be using a zillion custom library functions do work with basic data types rather than the standard abstractions like in Scala - so to really understand the line of Java you also need to grok the method its calling); however grokking the multiple statements which represent a single line of Scala is usually much harder. i.e. the the bigger picture, the intents of what the developer wanted are harder to grok in Java IMHO. Its not about understanding one line; its about grokking what the method does, how and why - understanding the whole block of code as one thing.

Using more powerful, concise notation to describe intents is always going to be harder to understand when compared to a trivial Java statement (variable.method() with some anonymous inner classes) - but I don't think thats the be all and end all - the bigger picture is way more important.

Understanding the intents and avoiding common silly bugs doing the common operations (iterating, filtering, copying, transforming data structures, having properties with optional lazy evaluation, navigating object structures etc etc) using custom hand-crafted code rather than reusable language features & patterns far outweighs how easy it is to understand a single isolated line of code.

Incidentally understanding a line of Scala code is pretty easy once you get used to functions/closures (it can take some time to get your head around defining types themselves, but the actual code inside a method is pretty trivial to understand).

Pretty much every language apart from Java now has functions/lambdas/closures in some form; many have for comprehension too and folks read them just fine. I don't buy into this 'we need to keep the language dumb because the poor Java developers don't have brains capable of reading this stuff'. You never hear a Groovy/Ruby/C#/JavaScript guy complain he can't grok a function/closure.

Type inference in Scala is awesome. The problem is that without knowing the types, and they could be several dependencies away, it really isn't obvious what a piece of code means

Just hover over an expression in your IDE and it shows you the type. If you have too much nesting on a single line, hit "Introduce Variable" in your IDE, then hover over the variable.

especially if someone has gone and overloaded some operators.

Just alt-click on the operator and your IDE takes you to the definition of the operator on the type; or can perform a quick scaladoc lookup describing what the operator means. Its just the name of a method. Its really no biggie.

My experience with Scala has been similar to many of those expressed in the comments. The documentation when it comes from the academic side of the world is completely foreign.

Please try reading one of the books I mention in my blog post - they are not at all academic and a very easy read!

The problem with Scala, however, is that it needs to support Java and the compromises are just too large

What compromises are those? Are they not just what any language which supported Java bytecode would have to do? (null is the biggest issue for me btw).

The compromises mean that Scala doesn't feel consistent. In striving to bridge both the imperatives and functional worlds, Scala loses some elegance

Elegance is bound to be subjective; I find Scala to be very elegant. Going with pure a functional language (or a pure OO language) loses something; there is value in OO and functional and I think Scala has unified the two concepts very well in a relatively simple, elegant language IMHO. The one trade off for this is a bit more complexity in the type system over a straight OO or functional language; but I firmly believe its worth it

Inspired by your recent comments and the general buzz, I've been learning Scala. I looked for a post of an analog to Groovy's "safe navigation", but I couldn't find one, so I wrote my own. James Iry posted a much better technique in a comment though:

Its not about understanding one line; its about grokking what the method does, how and why - understanding the whole block of code as one thing.

I agree that it is about understanding the code as a whole and brevity can aid this. The problem is that with the flexibility Scala offers, there are just more permutations to understand. Operator overloading is a good example of this. More on that later.

I don't buy into this 'we need to keep the language dumb because the poor Java developers don't have brains capable of reading this stuff'. You never hear a Groovy/Ruby/C#/JavaScript guy complain he can't grok a function/closure.

That's definitely not what I was getting at. But reduced to absurdity, that argument says that no language is too complex. I'll assume for the sake of the argument a line should be drawn somewhere (somewhere before Malbolge for example). In that case, I'd argue for a language that reduces the number of surprises or encourages a low WTF per minute ratio, to misquote Ward Cunningham. Obviously you can write complex code no matter what the language but again, some languages make it easier than others. It feels to me like Scala chooses power and flexibility over safety. Operator overloading is a good example of this. And again, the flexibility increases the mental overhead in understanding someone else's code, by way of example, there are at least 4 ways to curry a function in Scala.

Just hover over an expression in your IDE and it shows you the type

Great stuff. The Scala Plugin wasn't that smart before.

Just alt-click on the operator and your IDE takes you to the definition of the operator on the type

The problem is that you need to on your guard all the time. Suddenly there are no operators you can rely on. In practice it's probably not that large an issue but as autoboxing has demonstrated, not being able to rely on operators can cause subtle bugs...

What compromises are those?

Nulls, as you mentioned. Mutable state without Haskell's IO m-word "Here be dragons" to wrap around them. Verbosity when compared with some other functional languages (e.g. pattern matching or Algebraic Data Types).

Are they not just what any language which supported Java bytecode would have to do?

Possibly. I think that Scala could do more to discourage things like nulls or mutable state. Personally, I'm not a big fan of Lift and it put me off Scala somewhat specifically because, despite being a functional language, increased the amount of surprises I expect from a web framework, rather than reduced them. See "RequestVar" and "S".

In summary, I definitely believe there is a place for Scala. I agree that there is a growing need for Java to evolve or for something to replace it. If something is going to replace it, I would like it to be statically typed (we have a few good options for a Dynamically typed alternative already). It would be very good if whatever replaced it made dealing with concurrency easier. I don't believe Scala is going to be that replacement because I think it feels like the focus is on power and flexibility at the cost of safety. That said, I would be happy if it does become the replacement. An alterative web framework would be nice, though. Are the Grails guys busy?

Scala is a hybrid Object-Oriented/Functional Programming language on the JVM. When I heard that Twitter was using Scala, I was curious and started collecting all the sites and articles to learn scala programming. If you are interested check the link below for the big list I have gathered (more than 200 sites) for learning scala programming.http://markthispage.blogspot.com/2009/06/more-than-100-sites-to-study-scala.html

> There's too much temptation to use too much magic, and the infinite freedom to define symbolic operators absolutely terrifies me.

As someone who routinely writes mathematical code, I can attest that a _lack_ of operator overloading can make a program unreadable, difficult to maintain, verbose, and idiosyncratic.

For example: (a*a + b*b).sqrtprobably calculates the hypotenuse of something. But what does: a.multiply(a).plus( b.multiply(b) ).sqrt()do?

Using operator overloading to make code clear and easy to understand is generally a matter of following three principles:1. Use normal mathematical operators for conventional math.2. Use obviously weird looking operators for things that are not conventional math but have math-like properties.3. Use operators sparingly: they are hard to read, so use words instead of short sequences of symbols if there are many things to remember.

Operator overloading, it seems to me, gets people in trouble because they aren't taught simple principles like these. We aren't terrified by the infinite freedom to use letters (and even some numbers!) to construct variable and method names, even though people can define their methods as a0(), BQ(), c(), a00() and their variables as a1, a2, a10, a11, a12, a20 (and have different types on each). Instead, we teach people to not do that.

A little off-topic but have you considered the possibility of a Scala->JavaScript compiler? Google has done a Java->JavaScript compiler for GWT. I love GWT but would also love the opportunity to write Scala code instead of Java. I wonder how hard this would be, and how elegantly JavaScript could capture many of the Scala constructs, for instance, JavaScript supports closures. With Scala most of the magic is in the compiler isn't it?

The other thing is that with GWT Google have seemed to produce more compact/efficient code than hand written JavaScript. Since Scala is even more terse than Java, I wonder how compact the final result could be!