True Scala complexity

Update 2: Sorry for the downtime. Leave it to the distributed systems guy to make his blog unavailable. Nginx saves the day.

It’s always frustrating reading rants about Scala because they never articulate the actual complexities in the core language.

Understandable—this post is intended fill that gap, and it wasn’t exactly easy to put together. But there’s been so much resistance to the very thought that the complexity exists at all, even from on up high, that I thought it would be constructive to provide a clearer illustration of why it’s real and how it manifests itself.

So, here goes yet another Scala complexity rant, from someone who labels himself as a Scalaadvocate. And since this tends to be an insanely touchy subject for some lonely people, please read every sentence as implicitly beginning with “In my opinion…” before you fire off those death threats.

I’ve been hacking in Scala for a while now, since 2006. This was a time when exceptions from the compiler were routine, the standard library was in poor shape, the website was clearly designed by busy academic PL researchers, and the community felt quite a bit lonelier. (I was a much bigger masochist back then.)

Before that I had come from Haskell, the Lisps, etc., so I wasn’t a stranger to functional programming or category theory or what have you. Yet of all the languages I’ve picked up, it’s Scala’s growth that has been the most fascinating to spectate. I watched as this little research language—which I’ve never had any illusions of seeing “go mainstream”—slowly clawed its way out of obscurity, facing seemingly unusual amounts of bashing at every turn, all while believers cried “FUD!” and Martin patiently pushed Scala forward.

I often see the following issues raised in those Scala bashings, and this list includes Typesafe’s top priorities. My two cents are thrown in:

Performance: Yes, I’ve been bitten by for loops and whatnot. Fine, I can’t have my cake and eat it too. For those to-the-metal hotspots, I have to drop down to ~Java-level verbosity. I can live with that. Perhaps soon, compiler and VM optimizations will let us eat our cake as well.

IDE support: There’s no shortage of glitches, but it’s shaping up at a steady pace and is already plenty helpful.

Learning curve: This can be steep depending on where you’re coming from, but the resources and community support are all fairly solid and improving. That said, this is the closest issue to the subject of this post.

Build times

Binary compatibility: As of 2.9, sanity and predictability has been introduced into the versioning scheme.

Frameworks

Community

SBT: “SBT is confusing as shit.” –Questioner at the recent Scala SF meetup (and plentyothers, in different words). Amen, in so many ways. Thank heavens Typesafe has acknowledged this and is working on it.

Immature library ecosystem: Unsurprising for a relatively new language, but you do have the entire universe of Java libraries available.

WTF is /:? Why do I need to consult a periodic table to make HTTP requests? Those are unfortunate names/designs. Fortunately, it seems library authors, including the Scala team, are walking away with a healthy lesson learned, so we’ll probably see more accessible names in libraries in general.

Important issues to be sure, but these aren’t the ones keeping me up. They’ll take time and perspiration to fix, but hopefully with enough of both they’ll come.

What really bothers me is in the core language itself, something much harder to fix. The thing that Typesafe’s agenda does not touch on is the complexity of the language. If anything, Typesafe’s goals of driving Scala adoption, maintaining compatibility, etc. will likely slow down language evolution, and the less likely we are to see any fundamental course-corrections as a result.

What complexities am I talking about, exactly? This is one of those things that needs a “show, don’t tell” treatment. Complexity is a tricky thing to articulate, so no matter how many different ways I cut it, it’s far more effective to begin by just rubbing your face in it a bit.

So let’s take a little stroll through Scala programming land. And while we’re there, keep in mind that the problems encountered aren’t contrived cases I cooked up to make this argument; these are all (simplified versions of) situations we actually encounter while building real products.

Operating On Collections

(I’ve tried to make this as accessible as I could to non-Scala developers who perhaps have enough experience across languages to guess their way through the syntax, but you’re probably only going to really appreciate the frustrations if you have a passing familiarity with Scala basics. If you are a Scala developer, you should try to solve the problems we run into on your own first!)

Let’s make a super-simplified model of some basic collections we have in Scala, ignoring generics as a first step. We’ve got the primitive Array type from Java, and we have our nice Seq type from Scala. head is one of the many Scala collection methods, in Seq but not in Array. We’d like to pimpenrichArrays to Seqs so that we can call methods like head on Arrays.

This next function is an implicit conversion, so called because it’s automagically invoked whenever we have an Array and treat it like a WrappedArray (and thus a Seq). This auto-adaptation is called “enrichment.”

That’s basic stuff. Now we want to extend Seq with our own methods, such as ident (a dummy that’s just the identity function). We implicitly convert from Seq to an anonymous class containing our extension method:

Nope. To move forward, we can use higher-kinded types (Function[I[_], ...]), and instead of view bounds (which would give us the unhelpful error: type I takes type parameters), we have to take the converter in as an “explicit” implicit parameter in a curried argument list such that the type inference can flow from the first argument list to the type parameter list to the second argument list:

ident was just a placeholder, of course—otherwise we wouldn’t need to constrain it to be a Seq. We actually want a real method here—say, runs, which returns the number of runs of contiguous groups of elements, where groups are defined by the given predicate for adjacent elements. E.g., Seq(1,1,1,2,3,3) runs (_==_) should return 3. Here, for simplicity, we’ll just always return 0.

Oops. By using/returning new { ... }, we were actually using refinement types, which won’t work because on the JVM, where we have type erasure: Scala generates a single generic implementation of the isMajority method, and it calls methods on refinement types via reflection, so it has to fill in something for the ? in seq2richseq(xs).getClass.getMethod("isMajority", Array(?)), and we don’t know what that is. Whereas for runs, we actually do know what it is—this time, because of type erasure, we know it’s just a (_,_) => _ or Function2[_,_,_]—that it’s a Function2[A,A,Boolean] doesn’t matter. Did you get that? Here’s a more long-winded explanation.

Furthermore, we’ve all along been imposing a significant performance penalty by using reflection.

The solution is simple: introduce some boilerplate by hoisting the code out into a named type.

Dah, subseq2richseq is now further hindering us. For now let’s remove it, knowing that its removal prevents us from converting subtypes of Seq of arbitrary shape. Resetting our REPL to re-define everything except for subseq2richseq, we can make things work:

What was happening earlier was that we were giving away the type of A based on the rest of the expression after the implicit conversion—isMajority(Char) constrains A to be a Char. runs has no such luxury. So while the inferencer is unable to peek into the second argument list or the type bounds, it does peer beyond the entire implicit conversion at the subsequently invoked method. Anyway, you’ll just have to explicitly convert.

Now things get interesting.

So far we’ve been working in our tiny parallel universe, working on a cartoon of the collections library. We want to bring things into real Scala now, rather than reinvent the entire collections library. Let’s say we want to add a filterMap method: given a bunch of A and a function A => Option[B], return a collection of B.

To get a taste of how a proper collection method behaves, consider map, which is similar to filterMap. It’s defined in the base trait TraversableLike and has the signature:

We always return the most specific type we can. This works because of the functional dependencies in collections, CanBuildFrom[From, Elem, To]. For those new to Scala, you can learn more about how collections work from the collections architecture docs or from the innumerableStack Overflowquestions on the topic, but in short: CanBuildFrom[From, Elem, To] are these global helper objects that are passed as implicit parameters into collections method such as map and filter to allow them to construct a collection of the same (or most similar) type.

Let’s try to enrich collections with filterMap.

Which of the following is (are) the right signature(s) for the implicit conversion and for the filterMap method, if any? If you’re looking for easy eliminations, sorry: I have seen the equivalents to nearly all of the following lines on Stack Overflow, the mailing list, IRC/pastebins, or IRL. Many of these can actually get you somewhere, while not fully solving the problem as we defined it; hence, unfortunately, we see CanBuildFrom misused in manycases. So if you’re confused, you’re most certainly not alone.

The only way for you to have any hope of navigating these waters is to really understand how Scala’s type solver and implicit resolution algorithms work. And not just a little bit. You have to know these well enough to know what their limitations are. Otherwise, you’re going to be tossing things at the wall and seeing what sticks first, which can be infuriating and still wrong.

This might be somewhat alleviated if it weren’t for the fact that Scala’s type inference is intentionally not spec’d, or at least woefully underdocumented save what little you can glean from section 6.26.4 of the language spec. You can turn to all the books or webresources in search of definitive sources of information, but you won’t find it; the best you can do is to scavenge nuggets of insight here and there on Stack Overflow or the mailing lists and try to piece things together into a more coherent picture.

The answer to our original question? It turns out none of these are correct. In fact, it is impossible to insert a new method that behaves like a normal collection method. This, despite the heavy advertising of enrich my library. (Update: after others took various stabs at this problem, Miles Sabin got a solution that I believe comes the closest; it still requires per-source-type/shape boilerplate, but is able to factor out the filterMap definition. Why does the per-source-type/shape boilerplate make a difference? Because whenever you introduce a new type that can be enriched into a Seq or introduce a new enrichment for Seqs (say, across different libraries), you end up having to manually augment every such possible pair with the proper boilerplate. Anyway, this isn’t the point of this post, but I thought it was worth highlighting that snippet for the Scala users reading this.)

OK, what have we accomplished? We’ve kinda-sorta added a few extension methods to some collections types. We’ve also managed to stroll right through level L3, officially the most advanced tier of Scala language mastery. We’ve been wrangling implicit parameters, type bounds, view bounds, implicit conversions, higher-kinded types, functional dependencies, refinement types, and generalized type constraints, to name a few concepts. This is all just a taste; there are plenty more concepts—and more importantly, interactions among concepts—in Scala for us to bump into. Yet at the end of the day, our method still fails to behave like a normal collection method. But that isn’t the point.

How does this play out in the real world? Most people probably simply don’t bother with much of the above, just going with something like:

and being done with it. I’m betting this is closer to how Scala is predominantly used today, beyond the community of type-level metaprogramming junkies: most developers manage to avoid these questions altogether, giving in to clumsy code, opting for simplicity and getting things done, while remaining fully aware of the fact that their code is not as Scala-ish or as generic or as implicit or as type-safe as they ideally would like, all due to the very real complexity barrier.

Martin acknowledges many of the problems with Scala, and those are the ones Typesafe is actively addressing. But the one word he doesn’t seem to like using to describe Scala is “complex.” On the contrary, one might take him to sound downright dismissive:

Scala is used in a large and growing number of enterprises, some of them with more than 100 devs working with it. These people just get on with the job (and love it for the most part); they don’t find Scala’s “complexity” too daunting.

Martin’s right. Plenty of devs—my team included—are using Scala and getting on with it. We do like many things about it. We certainly don’t find it too daunting to be productive with. But that’s not to say it’s without complexity or frustration. There are even larger teams being productive with C++ as well, doing amazing things with it on which Scala only wishes it were brought to bear…but enough of the world has already made my point for me on C++.

There also appears to be a variety of other ways of deflecting this criticism:

Establishing multiple tiers of Scala usage, under the pretense that one can operate in isolation within any particular tier without being exposed to concepts from the upper tiers.

Reminding us that Scala at its essence consists of just a few powerful orthogonal features; in this view it’s simpler than more ad-hoc designs such as Java’s. Typical response when confronted with the complexity question:

By a lot of objective measures, Scala does not have a lot of features. For instance, it has fewer key words than many other languages including C#, C++, and even Java. It has its grammar size, so if you look at the size of the context free grammar, that also is about the same size as Java 5, slightly smaller and certainly much, much smaller than a grammar like C#, C++, or things like that.

Assuming that what developers object to is foremost the non-trivial set of foreign concepts:

It’s true that there’s a lot in the language, in particular a lot of very fundamental concepts, which take a lot of time for some people to learn and to absorb, things like traits, something that is new in Scala, pattern matching, the whole functional aspects that you have of closures and functions as parameters is new for many people and so on.

To be fair, the criticism isn’t exactly focused; I’ve heard complaints as far-ranging as, “Oh my god the constructor arguments are next to the class name. Shit is way too complex.” And our BDFL is right on many counts; he doesn’t lie when he asserts that at its core Scala is a language with a smaller number of orthogonal features than found in many other languages. As presented in the books and tutorials on Scala, everything seems manageable. However, the problem is that each feature has substantial depth, intersecting in numerous ways that are riddled with nuances, limitations, exceptions and rules to remember. It doesn’t take long to bump into these edges, of which there are many. These are not issues with Scalaz, and these are not things you can shield yourself from by constraining yourself to the L1/A1 expertise tier. I, along with every other developer I watched pick up Scala, became intimately aware of the issues right off the bat. There’s a reason why awholecottageindustry of “better Java” languages has emerged—languages that deliberately forego the sophisticated levels of expressiveness captured by Scala.

I also don’t buy that the individual concepts in Scala are what’s stumping smart programmers. Traits, pattern matching, closures, higher kinds, typeclasses, functors, monads, etc. might induce a paradigm shift and require some digestion time if you’ve only ever learned Java, but these aren’t inherently over-complex. I’m saying Scala—that particular recipe for mixing all these concepts—is complex. Haskell, for instance, has a rich ML-style type system and shares a great deal of those concepts in common with Scala, but once you’re in it, Haskell is significantly simpler to use, whereas Scala has many more concerns to juggle: object orientation/subtyping, implicits, Java interop, and much more.

And what do I mean by “complex,” anyway? Complexity is just a measure; isn’t a threshold always relative? Look: at the end of the day, features like an expressive type system are there to make us more productive, whether it’s aiding us in the construction of correct programs, or allowing us to concisely express our abstractions, or keeping our programs maintainable over time and across developers. But we have only so much patience for dicking around with types, since the returns diminish beyond a certain point. So, by Scala complexity I mean the degree to which the features in Scala are intricately intertwined, laden with surprising rules and exceptions. And when I say Scala is complex, that’s short for saying that I believe idiomatic Scala approaches “too much complexity to bother with” status for the majority of (competent and capable) developers out there; it has certainly already crossed that line for others, who have chosen to not use the language.

During a lunch I had with Guido van Rossum a while back, he explained to me something very important—that “language design is not just about solving puzzles.” Think what you will of Python (I can already sense all the statically typed FP die-hards turning their noses up at the mere mention), but the assertion is undoubtedly true, and it’s what makes language design hard. It’s not just about figuring out ways to fit in as much functionality as possible or finding the smallest number of primitives that maximize expressive power; language design is just as much a usabilityproblem. The feeling I get from Scala is that such concerns take a back seat to squeezing as much as possible out of the feature set and its interactions. These two objectives can be made to align, but that’s hardly a given.

I’m not exactly sure where to go from here. Some of the problems I highlighted are tractable and will hopefully be improved in future versions of Scala, with either compiler improvements or updated language design choices. But other problems are simply hard problems, the solutions to which are unclear, at least not without giving up the very same powerful features that make Scala so malleable and attractive. For instance, one can’t simply introduce a combinatorial explosion into the search space for the type solver/implicit resolution algorithms; Scala builds take long enough already!

What is certain is that Scala complexity is quite real. While I wouldn’t doubt that the smart people comprising Typesafe are fully aware of that, they certainly don’t show it or at least any concern over it. Acknowledging problems matters: it assures the world that you understand the problems, and opens the path (and the communication) toward solutions.

Beyond Typesafe, the denial pervades the Scala community. There’s no shortage of broken analogies and shallowarguments dismissing the complexity concerns altogether or arguing that it doesn’tmatter. What’s worse, something I see far too often from the community is the insistence that anyone proclaiming Scala complexity is just incompetent or spreading FUD. All this behavior does the community a disservice; it’s a sure way to alienate talent and to prevent problems from being fixed. We need to be able to open our eyes wide enough to critique the status quo in order for progress to be made.

Does all this mean I’ve given up on Scala? Hardly. No tool is perfect, and any engineer worth her salt should have a clear understanding of both the pros and the cons of the tools at her disposal. If you’ve been using a language for a while and can’t rant about it, that’s probably a bad sign.

Notice how several of the problems I pinpoint above are of the form, “How do I accomplish this expression that isn’t even possible in many other languages?” And these are mostly static build issues, which are far from the worst fates imaginable. Certainly, if you don’t care at all about clumsy code or resorting to escape hatches or “writing Java in Scala,” it’s frequently possible to bang out mostly-type-safe Scala while side-stepping battles against the compiler straitjacket. Plus, let’s not lose sight of all the things Scala brings to the table, none of which I’ve described. If there’s one thing more time-consuming than illustrating the complexities of the Scala language, it’s illustrating all the advantages, and I’ve already spent more time on this post than I’d care to admit.

I’ve never studied Scala and am in love with Lisp-y languages because of their simplicity. So obviously I am learning / using Clojure these days. Since you already know Lisp, would you mind explaining why you chose Scala? Maybe I am an ignoramus but I fail to understand why would anyone pick Scala over Clojure?

alforsan

I also don’t buy that the individual concepts in Scala are what’s stumping smart programmers. Traits, pattern matching, closures, higher kinds, typeclasses, functors, monads, etc. might induce a paradigm shift and require some digestion time if you’ve only ever learned Java, but these aren’t inherently over-complex. I’m saying Scala—that particular recipe for mixing all these concepts—is complex. Haskell, for instance, has a rich ML-style type system and shares a great deal of those concepts in common with Scala, but once you’re in it, Haskell is significantly simpler to use, whereas Scala has many more concerns to juggle: object orientation/subtyping, implicits, Java interop, and much more.

vero

The data with tips really useful if you ask me. Thanks
self-same a great deal before hand.

Good , your blog liked it has very good information I commented that just started a blog similar issues and I wonder if you could stop by the blog and give me some other advice thanks for attention.???? ???????

Mrs Jones did not have a husband, but she had two sons. They were big,
strong boys, but they were lazy. On Saturdays they did not go to school,
and then their mother always said, ‘Please cut the grass in the garden
this afternoon, boys.’ The boys did not like it, but they always did it.

friends you are now with independent female at ahmedabad to taking a better female service at ahmedabad city. At this moment you are at right palce because this a well famous and high in class of quality. This all shown by my clients .http://fizza.in/

haye guys this is my tendency that I ever give full respect with my clients they give their better response with me they take regular tuch with me using social media. I am also active at social if I found me free they tell me thinks to improve some mistake.escort service in surat