It's a very detailed email from an employee at Yammer on why they are
migrating from Scala back to Java. I know it was intended to be private,
but it's gone public now, and there's nothing personal in the email anyways.

It's a very detailed email from an employee at Yammer on why they are migrating from Scala back to Java. I know it was intended to be private, but it's gone public now, and there's nothing personal in the email anyways.

I can't comment usefully on all of the points, but as someone who attempts to do a lot of high-performance work in Scala, they've captured the issues with high-performance Scala reasonably well. In detail:

1. Don't ever use a for-loop. Creating a new object for the loop closure,
passing it to the iterable, etc., ends up being a forest of invokevirtual calls,even for the simple case of iterating over an array.

Indeed. For loops are slow. After "create the pipe operator for yourself", this is probably the second-most-common question/answer pair on StackOverflow. JVMs cannot reliably optimize even specialized range code in for loops, especially if they're nested.

2. Don't ever use scala.collection.mutable. Replacing a scala.collection.mutable.HashMap with a java.util.HashMap in a wrapper produced an order-of-magnitude performance benefit for one of these loops.

This is also true in some cases. The Scala HashMap does an embarrassing amount of boxing of key-value pairs. For example, the += method takes a tuple (often created by the user to begin with), then accesses the two elements, packaging them into an Entry object, which it finally passes off to HashTable. For simple mappings this can give a ~10x slowdown. I wouldn't say to _never_ use scala.collection.mutable (e.g. ArrayBuffer is pretty close to java.util.ArrayList in performance), but there are some gotchas lurking in there, with HashMap being one of the worst offenders.

3. Don't ever use scala.collection.immutable. Replacing a scala.collection.immutable.HashMap with a java.util.concurrent.ConcurrentHashMap in a wrapper also produced a large performance benefit for a strictly read-only workload. Replacing a small Set with an array for lookups was another big win, performance-wise.

Here, I disagree. What is your use case, and how sure are you that you are _actually_ not modifying that java.util.concurrent.ConcurrentHashMap anywhere? You can often write less-formally-safe code that performs better, but that doesn't make it a good idea unless you identify this as a key bottleneck, at which point you fix the key bottleneck and take away the lesson that _you fix key bottlenecks_, not that you _always use less-formally-safe code_.

4. Always use private[this]. Doing so avoids turning simple field access into an
invokevirtual on generated getters and setters. Generally HotSpot would end upinlining these, but inside our inner serialization loop this made a huge
difference.

That's not nearly as big of an issue as always using final val in classes if it can in fact be final (even in places where it seems redundant, such as inside a class itself marked final). Anyway, using private[this] is a good practice for other reasons: it prevents classes from messing with each others' internals, which makes it harder to reason about the code.

5. Avoid closures [...] we stopped seeing lambdas as free and
started seeing them as syntactic sugar on top of anonymous classes and thusacquired the same distaste for them as we did anonymous classes

This one I also disagree with. Understand the relative cost of closures compared to the code that they enclose. Use appropriately. Also, in deeply nested loops, create the functions by hand at the outer level; the compiler doesn't notice that it can move closure creation into the outside loop. For example, if you have an array of options of strings, using arr.foreach(_.foreach(x => ...)) takes about twice as long as arr.foreach(_.foreach(f)) where f is defined outside.

Anyway, these criticisms seem, for the most part, reasonably well-founded, except that the bottom line is that you miss out on many of Scala's best features in your critical inner loops. Unless your entire code base is critical inner loops, using Java is unlikely to make your life easier. (There is a lot more to remember when writing high-performance Scala than when writing high-performance Java, however.)

When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.

Also, in my opinion, the critisims layed against SBT is entirely justified. I think the idea of having a Scala based build tool is great, and the incremental compilation is great, and the console is great. However the custom DSL is extremely hard to use, and the implementation source code is impossible to read. Additionally many features just do not work as expected, either by design or due to bugs.

Just to be very clear, there are no "for loops", are you talking about for comprehensions or just "foreach"?

were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.

Also, in my opinion, the critisims layed against SBT is entirely justified. I think the idea of having a Scala based build tool is great, and the incremental compilation is great, and the console is great. However the custom DSL is extremely hard to use, and the implementation source code is impossible to read. Additionally many features just do not work as expected, either by design or due to bugs.

When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.

That's simply not true. The position is that we should not do isolated ad-hoc fixes but go to the root of things. Scala's overall optimizations need to be good enough that they can optimize standard for-loops. And they are already very close of doing this. I wonder whether Yammer ever tried compiling with -optimize.
Cheers -- Martin

Just to be very clear, there are no "for loops", are you talking about for comprehensions or just "foreach"?

You are not a compiler, you are a human being who can infer from the context. So you know EXACTLY what I mean. Your question serves no useful purpose but adds the flames.

What are you talking about, I asked a perfectly sensible question.Are you talking about optimizing foreach or are you talking about something more general when it comes to desugaring for comprehensions, you definitely don't need to get all ballistic for no reason.

Maybe I'm too used to scala-user... didnt realize this was on debate. I am respectfully taking myself out of this dicussion.

I used to religiosly compile all my code with -optimize. However at some point I found that it was inlining excessively and resulted in slower bytecode. I have not tried -optimize for a long time. My understanding is that if -optimize worked so well, it would be enabled by default. If it's not the default, there are good reasons for it.

When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.

That's simply not true. The position is that we should not do isolated ad-hoc fixes but go to the root of things. Scala's overall optimizations need to be good enough that they can optimize standard for-loops. And they are already very close of doing this. I wonder whether Yammer ever tried compiling with -optimize.
Cheers -- Martin

Aleksey Nikiforov skrev 2011-11-30 21:27:
> When I first dicovered that for loops were not optimized I was surpised.
> When I discovered a resistance to optimizing them I was shocked. The
> decision to do nothing about for loops baffles me to this day.

For comprehensions are just syntactic sugar. Optimizing just them is not
very useful, instead effort should be put into generic optimization of
HOF's (and other inlining).

I used to religiosly compile all my code with -optimize. However at some point I found that it was inlining excessively and resulted in slower bytecode. I have not tried -optimize for a long time. My understanding is that if -optimize worked so well, it would be enabled by default. If it's not the default, there are good reasons for it.

I assume that compiling with -optimize takes longer, and that is why it is not the default. If it takes no longer to compile (and it works reliably), then yes it might as well be the default.

When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.

That's simply not true. The position is that we should not do isolated ad-hoc fixes but go to the root of things. Scala's overall optimizations need to be good enough that they can optimize standard for-loops. And they are already very close of doing this. I wonder whether Yammer ever tried compiling with -optimize.
Cheers -- Martin

On 30 Nov 2011, at 18:24, Jeff Nowakowski wrote:
> I'm surprised I haven't seen this mentioned by now:
>
> Email critique of Scala: https://gist.github.com/1406238
>
> A sample new story on it: http://www.infoq.com/news/2011/11/yammer-scala
>
> It's a very detailed email from an employee at Yammer on why they are migrating from Scala back to Java. I know it was intended to be private, but it's gone public now, and there's nothing personal in the email anyways.

The point that he makes that (to me, at least) resonates most strongly relates to the Scala community. I don't know how to fix it, but we really do need to fix it.

It must be possible to create a supportive and welcoming environment for people coming to the language. Right now, the Scala community is nothing like that.

We've been using Scala heavily over the last 18 months and are in the process of selecting which language we'll use for a major new development. There are two things that count strongly against Scala - the binary compatibility issue and the poisonous nature of community that surrounds the language.

Personally, I'm strongly in favour of Scala. I'd rather stick pins in my eyes than use Java and I don't see any credible choice other than Scala. But my case is severely weakened by by the squabbles, flame wars and just plain rudeness that constantly breaks out here.

How do we (can we) address this?

--
paul.butcher->msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

On Wed, Nov 30, 2011 at 10:37:02PM +0100, √iktor Ҡlang wrote:
> > You are not a compiler, you are a human being who can infer from the
> > context. So you know EXACTLY what I mean. Your question serves no useful
> > purpose but adds the flames.
> >
>
> What are you talking about, I asked a perfectly sensible question.
> Are you talking about optimizing foreach or are you talking about something
> more general when it comes to desugaring for comprehensions, you definitely
> don't need to get all ballistic for no reason.

If you needed clarification because you were confused then Aleksey
wrongly took offense at your reply. If you were taking the opportunity
to snipe at him for using slightly inexact terminology then his
reaction would make sense.

Martin (and most of us I think) understood what he meant, and Martin
referred to a "standard for loop" in his reply. I don't think anyone
would reply to Martin Odersky pointing out the fact that Scala doesn't
have for loops and asking for clarification.

Online, it's easy to perceive hostility online when none exists. It's
also easy to be casually rude to people. Let's all try to lighten up a
bit.

When I first dicovered that for loops were not optimized I was surpised.
When I discovered a resistance to optimizing them I was shocked. The
decision to do nothing about for loops baffles me to this day.

For comprehensions are just syntactic sugar. Optimizing just them is not very useful, instead effort should be put into generic optimization of HOF's (and other inlining).

/Jesper Nordenberg

This is exactly the thinking that baffles me. There are for-loops in Java. Not comprehensions, LOOPs. Anyone coming from Java will expect for LOOPs or simlar functionality.

It does not matter to me if it is a syntactic sugar or if they are comprehensions. What matters to me is to be able to write the code:

var sum = 0; for (i <- 0 until array.length) sum += array(i)

And have it work as fast as possible. Scala fails that expectation. It fails it so much that this is one of the most frequent questions on stack overflow. Also, it appears to be the number 1 on the list of problems with Scala as viewed by the employee at Yammer who wrote the email:

1. Don't ever use a for-loop.

So, they were looking for loops as well, not comprehensions. Why not send them an email and ask them if perhaps they were "talking about for comprehensions or just foreach"?! If you think I was overreacting, I would LOVE to see their response.

It is nice to have for-comprehensions optimized. But I really do not care about for-comprehensions. I want something that looks like for loops and works like for loops. 99% of my code uses emulated for-loops via while-loops. It's painful and error prone, so I take this issue very personally.

Not providing proper for-LOOPs is akin to saying: Scala is not meant for the applications that require for-loops. Which sends the message that Scala is not meant for performance-sensitive applications. So guess what, the folks at Yammer got the message and went back to Java.

You can tell me that they are comprehensions, or a more general case should be optimized. But I do not care about those uses at all. Ranges and closures can never be optimized to the level of while loops. And I need the LOOPs. And so do many other people not interested in functional aspects of Scala.

On 30 Nov 2011, at 18:24, Jeff Nowakowski wrote:
> I'm surprised I haven't seen this mentioned by now:
>
> Email critique of Scala: https://gist.github.com/1406238
>
> A sample new story on it: http://www.infoq.com/news/2011/11/yammer-scala
>
> It's a very detailed email from an employee at Yammer on why they are migrating from Scala back to Java. I know it was intended to be private, but it's gone public now, and there's nothing personal in the email anyways.

The point that he makes that (to me, at least) resonates most strongly relates to the Scala community. I don't know how to fix it, but we really do need to fix it.

It must be possible to create a supportive and welcoming environment for people coming to the language. Right now, the Scala community is nothing like that.

We've been using Scala heavily over the last 18 months and are in the process of selecting which language we'll use for a major new development. There are two things that count strongly against Scala - the binary compatibility issue and the poisonous nature of community that surrounds the language.

I find the part about the "poisonous nature of community" hard to believe. First, I didn't realize that it is any more "poisonous" than any other online community. And even if it were, I can't imagine that would be much of a factor in selecting a language for a business. Then again, I'm lucky enough to have the freedom to select Scala without any approval from anyone else, and I'm the only one in my immediate organization who currently uses it, so maybe I just can't see the problem you are referring to.

Personally, I'm strongly in favour of Scala. I'd rather stick pins in my eyes than use Java and I don't see any credible choice other than Scala. But my case is severely weakened by by the squabbles, flame wars and just plain rudeness that constantly breaks out here.

In defense of Victor, he did preface his question with "Just to be very clear, ...". I think the hostile reply was an overreaction, but that sort of thing is common in online discussions.

>> Personally, I'm strongly in favour of Scala. I'd rather stick pins in my
>> eyes than use Java and I don't see any credible choice other than Scala. But
>> my case is severely weakened by by the squabbles, flame wars and just plain
>> rudeness that constantly breaks out here.

i wonder to what degree Scala is powerful enough to let various
disparate communities think they can use it, and then they come to
loggerheads on the lists? Like, there's the "i wish i were doing
haskell" crowd that will poo poo you if you don't already know the
typeclassopedia by heart; there's the performance nuts; there's the
people who really just kinda want a slightly less painful java but not
much more than that; there's yadda yadda yadda.

with java nobody is under any mistaken impression that it is or can be
anything other than plain old java. if you add on something like guava
or apache functional stuff you know you are out in la la land and your
noize only shows up on those mailing lists for only like-minded
suckers to read.

When I first dicovered that for loops were not optimized I was surpised.
When I discovered a resistance to optimizing them I was shocked. The
decision to do nothing about for loops baffles me to this day.

For comprehensions are just syntactic sugar. Optimizing just them is not very useful, instead effort should be put into generic optimization of HOF's (and other inlining).

/Jesper Nordenberg

This is exactly the thinking that baffles me. There are for-loops in Java. Not comprehensions, LOOPs. Anyone coming from Java will expect for LOOPs or simlar functionality.

It does not matter to me if it is a syntactic sugar or if they are comprehensions. What matters to me is to be able to write the code:

var sum = 0; for (i <- 0 until array.length) sum += array(i)

Alright, so you're not talking about for comprehensions, you're talking about the foreach-method. Because optimizing desugaring of for comprehensions is a much deeper topic.

And have it work as fast as possible. Scala fails that expectation. It fails it so much that this is one of the most frequent questions on stack overflow. Also, it appears to be the number 1 on the list of problems with Scala as viewed by the employee at Yammer who wrote the email:

1. Don't ever use a for-loop.

So, they were looking for loops as well, not comprehensions. Why not send them an email and ask them if perhaps they were "talking about for comprehensions or just foreach"?! If you think I was overreacting, I would LOVE to see their response.

So you allow yourself to overreact because someone else might overreact even more?I'm trying to understand the topic at hand so there can be a meaningful discussion, this is scala-debate after all.

It is nice to have for-comprehensions optimized. But I really do not care about for-comprehensions. I want something that looks like for loops and works like for loops. 99% of my code uses emulated for-loops via while-loops. It's painful and error prone, so I take this issue very personally.

Something like that should be a prime use case for a macro, to expand it into a while construct, I mean.

Not providing proper for-LOOPs is akin to saying: Scala is not meant for the applications that require for-loops.

So your only problem with foreach is that it does not have the raw performance that you need? Otherwise the for-comprehension sugar is just fine? So "proper" in this sentence means "high performance".

The question is how deep the rabbit hole goes, what if you add a guard to a for comprehension, should the optmizations be applied anyway, because then we're just not simply talking about "for loops" anymore, and people might wonder why the "for loop" users have so much better performance than the "for loop with guard" users.
This is why I suspected that the discussion was about foreach, but it would really be about optimizing the desugaring of for comprehensions, which is why I needed to ask what we actually were talking about.

Which sends the message that Scala is not meant for performance-sensitive applications. So guess what, the folks at Yammer got the message and went back to Java.

When writing high performance code you _always_ have to use the right tool for the job, if that means dropping into a little bit of Java, or into sun.misc.Unsafe, into cache-line padding classes, write your own custom data structures or doing object pooling. But that Scala would be unsuited for high performance code just not true, I think 45 million asynchronous messages processed per second on a 6 core box is pretty decent performance.

You can tell me that they are comprehensions, or a more general case should be optimized. But I do not care about those uses at all. Ranges and closures can never be optimized to the level of while loops. And I need the LOOPs. And so do many other people not interested in functional aspects of Scala.

So essentially you'd just want Javas good old for(... ; ....; ...) ?As I said earlier, if nothing else, it could most probably be implemented using macro expansion in the future. So it is just a special sugar for a while construct.

>> Personally, I'm strongly in favour of Scala. I'd rather stick pins in my
>> eyes than use Java and I don't see any credible choice other than Scala. But
>> my case is severely weakened by by the squabbles, flame wars and just plain
>> rudeness that constantly breaks out here.

i wonder to what degree Scala is powerful enough to let various
disparate communities think they can use it, and then they come to
loggerheads on the lists? Like, there's the "i wish i were doing
haskell" crowd that will poo poo you if you don't already know the
typeclassopedia by heart; there's the performance nuts; there's the
people who really just kinda want a slightly less painful java but not
much more than that; there's yadda yadda yadda.

with java nobody is under any mistaken impression that it is or can be
anything other than plain old java. if you add on something like guava
or apache functional stuff you know you are out in la la land and your
noize only shows up on those mailing lists for only like-minded
suckers to read.

That I not true, I got flamed on SO for using obscure language features like instance initializers and qualified "this" when some guy wanted the golfiest snippet to traverse a list + this in pure Java:

Guys, this is on the edge of acceptable, even for Scala debate. Please "tone this down". I know it's email and no tone may have been implied, so let's be extra careful here and take a few seconds before any more responses.

>> Personally, I'm strongly in favour of Scala. I'd rather stick pins in my
>> eyes than use Java and I don't see any credible choice other than Scala. But
>> my case is severely weakened by by the squabbles, flame wars and just plain
>> rudeness that constantly breaks out here.

i wonder to what degree Scala is powerful enough to let various
disparate communities think they can use it, and then they come to
loggerheads on the lists? Like, there's the "i wish i were doing
haskell" crowd that will poo poo you if you don't already know the
typeclassopedia by heart; there's the performance nuts; there's the
people who really just kinda want a slightly less painful java but not
much more than that; there's yadda yadda yadda.

with java nobody is under any mistaken impression that it is or can be
anything other than plain old java. if you add on something like guava
or apache functional stuff you know you are out in la la land and your
noize only shows up on those mailing lists for only like-minded
suckers to read.

That I not true, I got flamed on SO for using obscure language features like instance initializers and qualified "this" when some guy wanted the golfiest snippet to traverse a list + this in pure Java:

> So essentially you'd just want Javas good old for(... ; ....; ...) ?
> As I said earlier, if nothing else, it could most probably be implemented
> using macro expansion in the future. So it is just a special sugar for a
> while construct.

When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.

That's simply not true. The position is that we should not do isolated ad-hoc fixes but go to the root of things. Scala's overall optimizations need to be good enough that they can optimize standard for-loops. And they are already very close of doing this.

By "very close" I assume you mean "works in many cases as long as the number of items is large"? I've never gotten good performance out of a for-comprehension on, for instance, short arrays or ranges. (Well, except for the ScalaCL plugin.) I think it's lingering object creation overhead...haven't chased it down.

Also, one can forgive people for not compiling with -optimize, since (1) it is slower, and (2) for years it did nothing beneficial for performance.

Anyway, let's get macro capability (or compile {} blocks) implemented and then everyone who wants to can add a jfor that works just like the Java for (or the collections library can be compilable).

Hi,
I don't know if that's what's planned but I guess that once the effect
system is there and once "until" is annotated as pure, some
deforestation or one of its variants could be applied there.
Paul

On Wed, Nov 30, 2011 at 22:35, martin odersky wrote:
>
>
> On Wed, Nov 30, 2011 at 9:27 PM, Aleksey Nikiforov wrote:
>>
>> When I first dicovered that for loops were not optimized I was surpised.
>> When I discovered a resistance to optimizing them I was shocked. The
>> decision to do nothing about for loops baffles me to this day.
>>
> That's simply not true. The position is that we should not do isolated
> ad-hoc fixes but go to the root of things. Scala's overall optimizations
> need to be good enough that they can optimize standard for-loops. And they
> are already very close of doing this. I wonder whether Yammer ever tried
> compiling with -optimize.
>
> Cheers
>
> -- Martin
>

Am Mittwoch, 30. November 2011, 23:08:57 schrieb Paul Butcher:
> On 30 Nov 2011, at 18:24, Jeff Nowakowski wrote:
> > I'm surprised I haven't seen this mentioned by now:
> >
> > Email critique of Scala: https://gist.github.com/1406238
> >
> > A sample new story on it: http://www.infoq.com/news/2011/11/yammer-scala
> >
> > It's a very detailed email from an employee at Yammer on why they are
> > migrating from Scala back to Java. I know it was intended to be private,
> > but it's gone public now, and there's nothing personal in the email
> > anyways.

First: I think it's to be expected that not every adoption will succeed. One
size doesn't fit all. Scala has it's issues (as a language, as an ecosystem,
as a community) but so does .

Furthermore it's a little unfortunate that a personal communication leaked
into the public. But now it's on the table for all to behold. And some of the
criticism is not really new (at least to me). Many issues are already being
adressed by typesafe so there is a chance for improvement.

My take home message: many (maybe even small) burdens and inconveniences have
summed up to a point where scala was not perceived as being worth the cost.
Fair enough.

> The point that he makes that (to me, at least) resonates most strongly
> relates to the Scala community. I don't know how to fix it, but we really
> do need to fix it.

I don't think we can fix it on this medium. It's the internet, it's all about
personality. It's uncontrolled.

And yes - some of the flames are used as evidence by people who don't like
scala to show that the community is hostile. Nothing can change that (unless
there are no longer such posts - but this won't happen. It's the internet).

So what can be done?

Officially: Documentation, education, official statements from the scala
maintainer about all major topics. Provide official references and resources.
But all this has already been done or is on its way.
So I think it's ok - scala is on its way.

Sometimes people don't want to be educated - not even remotely. They just want
a quick copy & modify & paste solution for an obstacle to get pressing
business done. Not providing a solution but trying to convince them that their
solution outline is crap firsthand or trying to baffle them with links to
fundamental articles of some sort won't scala do a favour. If I don't find a
switch to turn an engine on I don't want to read about electricity...

Providing a solution together with a hint that there is room for improvement
will often be received better.

What would be nice regarding performance is to have a page on the
scala site where:
- It is explained what you can expect from -optimize, on a server JVM.
- Advices are given to improve performance in bottlenecks.

That would help in situation like those described in the mail that
started this discussion.

On 30 Nov 2011, at 22:41, Russ Paielli wrote:
> I find the part about the "poisonous nature of community" hard to believe. First, I didn't realize that it is any more "poisonous" than any other online community.

Sadly, it really is. You're absolutely correct that misunderstandings and flame wars are endemic on the internet, but for some reason the Scala community is much more prone to them than any other I've ever participated in. Worse, we have a habit of unloading on newbies so that their first experience of trying to participate is a negative one. You don't have to look far to find many examples of exactly this.

I spent quite a lot of time using Ruby (from around 2006 to 2010). One of the things that really helped Ruby adoption was the fact that the community was welcoming and supportive. Scala would benefit hugely if we could achieve the same.

(and yes, I know that if you go search the archives, you will find examples of flame wars in the Ruby community - the infamous Zed Shaw falling out of love with Ruby being a classic example. But that doesn't change the fact that the overall impression of the community was positive, in stark contrast with Scala at the moment).

> And even if it were, I can't imagine that would be much of a factor in selecting a language for a business.

You have at least two examples of people telling you that it's a problem. Me, and the Yammer post that sparked this discussion off. And there are plenty of others.

Community is an *incredibly* important part of what makes a language successful.

> In defense of Victor, he did preface his question with "Just to be very clear, ...". I think the hostile reply was an overreaction, but that sort of thing is common in online discussions.

My post was not intended to be aimed at Victor at all (in fact, I hit "send" before his message arrived in my inbox). One of the unfortunate side-effects of the bad atmosphere that surrounds this list is that people get "jumpy" and start taking offence unnecessarily. It becomes a self-perpetuating negative cycle. We all need to participate in breaking that cycle.

--
paul.butcher->msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

Thanks for the pointer. Of course there are multiple ways to interpret the results, but (playing the devil's advocate here) one perspective is that message exchange has an upper bound of 45 MHz on a multi-core, multi-GHz machine.Cheers,- Tiark

My understanding is that if -optimize worked so well, it would be enabled by default. If it's not the default, there are good reasons for it.

Same here. If an optimization is stable, trustworthy, doesn't double compilation time, doesn't bloat the resulting code and speeds things up, it should be there by default. If it doesn't do all of these things then I probably don't have time to compile exactly those scala classes that will benefit from it under -optimize and the rest of my codebase without it.
I understand that it is a Hard Problem to have the compiler optimize away arbitrary for comprehensions over arbitrary structures fully. In the special case of desugaring Range to a for(init;test;incr), or List to a for(list;!=Nil;l=tail), for example, I think this is critical to a whole range of applications. When writing some matrix code, I had to manually desugar all my for comprehensions to get a 100x speedup, as all the time was taken in boxing and instantiating HOF instances.
How is the effects system coming along?Matthew

When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.

That's simply not true. The position is that we should not do isolated ad-hoc fixes but go to the root of things. Scala's overall optimizations need to be good enough that they can optimize standard for-loops. And they are already very close of doing this. I wonder whether Yammer ever tried compiling with -optimize.
Cheers -- Martin

Thanks for the pointer. Of course there are multiple ways to interpret the results, but (playing the devil's advocate here) one perspective is that message exchange has an upper bound of 45 MHz on a multi-core, multi-GHz machine.

That was not the upper bound, that was after a bit of tuning, not full tuning, on a 6-core box.This was just on a 6-core box, we're running tests on our 48-core box right now for additional optimization opportunities.

Right now we've found a bottleneck in java.util.concurrent.{ThreadPoolExecutor, LinkedBlockingQueue, ArrayBlockingQueue},so we're working to build our own stuff for that.

On Wed, Nov 30, 2011 at 20:11, Aleksey Nikiforov wrote:
> On Wed, Nov 30, 2011 at 3:43 PM, Jesper Nordenberg
> wrote:
>>
>> Aleksey Nikiforov skrev 2011-11-30 21:27:
>>
>>> When I first dicovered that for loops were not optimized I was surpised.
>>> When I discovered a resistance to optimizing them I was shocked. The
>>> decision to do nothing about for loops baffles me to this day.
>>
>>
>> For comprehensions are just syntactic sugar. Optimizing just them is not
>> very useful, instead effort should be put into generic optimization of HOF's
>> (and other inlining).
>>
>> /Jesper Nordenberg
>
>
>
> This is exactly the thinking that baffles me. There are for-loops in Java.
> Not comprehensions, LOOPs. Anyone coming from Java will expect for LOOPs or
> simlar functionality.
>
> It does not matter to me if it is a syntactic sugar or if they are
> comprehensions. What matters to me is to be able to write the code:
>
> var sum = 0; for (i <- 0 until array.length) sum += array(i)

I had noticed before this strange characteristic of Scala bytecode of
breaking loop bodies in two parts, and jumping between them. I suppose
that's probably related to the inlining of closures, but, still, this
unnecessary jumping can't possible be good.

Anyway, notice how lines 104-111 in Scala are the same as lines 14-20
of Java. Where it differs is that Scala is using IntRef all over the
place. That's sum stored as IntRef on slot 5, on the heap. And look at
lines 93, 95 and 108: slot 8 is "i", but it reads it on 93, stores on
slot 6 on 95, then do the arithmetic, and next restores it from slot 6
on 108! Slot 6 need not exist at all, and it could have just read from
slot 8 at that point!

>
> And have it work as fast as possible. Scala fails that expectation. It fails
> it so much that this is one of the most frequent questions on stack
> overflow. Also, it appears to be the number 1 on the list of problems with
> Scala as viewed by the employee at Yammer who wrote the email:
>
> 1. Don't ever use a for-loop.
>
> So, they were looking for loops as well, not comprehensions. Why not send
> them an email and ask them if perhaps they were "talking about for
> comprehensions or just foreach"?! If you think I was overreacting, I would
> LOVE to see their response.
>
>
> It is nice to have for-comprehensions optimized. But I really do not care
> about for-comprehensions. I want something that looks like for loops and
> works like for loops. 99% of my code uses emulated for-loops via
> while-loops. It's painful and error prone, so I take this issue very
> personally.
>
> Not providing proper for-LOOPs is akin to saying: Scala is not meant for the
> applications that require for-loops. Which sends the message that Scala is
> not meant for performance-sensitive applications. So guess what, the folks
> at Yammer got the message and went back to Java.
>
> You can tell me that they are comprehensions, or a more general case should
> be optimized. But I do not care about those uses at all. Ranges and closures
> can never be optimized to the level of while loops. And I need the LOOPs.
> And so do many other people not interested in functional aspects of Scala.
>

It's a very detailed email from an employee at Yammer on why they are migrating from Scala back to Java. I know it was intended to be private, but it's gone public now, and there's nothing personal in the email anyways.

I hope that some good will come out of this. For example, it's no good if a company has a coding rule that says "use var i = 0; while (i < n) { ... i += 1 } instead of for (i <- 0 until n)" That should be a standard optimization, not one that requires a special flag. I don't think it's rocket science to address this. Ditto with the performance of the hash map. It should not be necessary to use java.util.HashMap.

The angst about TraversableLike and builder typeclasses is understandable. I had to work through that when writing the collections chapter for Scala for the Impatient. I ended up advising readers to use Iterable or Seq, and to ignore all the other interfaces. I realize this will outrage someone somewhere. And yes, the operators have gotten a bit out of hand in the collections library. I put them all into a table, and the result wasn't pretty. (The problem isn't the operators, but their inconsistency.) There is a lot to like about the collections library, but it's not perfect. Perhaps it needs to get redone one more time, or perhaps Scaladoc needs to be smarter about giving a more user-centric view.

Some of the other issues raised surely resonate with most of us: IDEs,
sbt/Maven, version incompatibilies. That will take blood, sweat, toil,
and tears to work through. Eventually it will all work, but you can't
blame a user to be frustrated about it.

It's a very detailed email from an employee at Yammer on why they are migrating from Scala back to Java. I know it was intended to be private, but it's gone public now, and there's nothing personal in the email anyways.

I hope that some good will come out of this. For example, it's no good if a company has a coding rule that says "use var i = 0; while (i < n) { ... i += 1 } instead of for (i <- 0 until n)" That should be a standard optimization, not one that requires a special flag. I don't think it's rocket science to address this. Ditto with the performance of the hash map. It should not be necessary to use java.util.HashMap.

Absolutely right. That's two things we will address in 2.10.

The angst about TraversableLike and builder typeclasses is understandable. I had to work through that when writing the collections chapter for Scala for the Impatient. I ended up advising readers to use Iterable or Seq, and to ignore all the other interfaces. I realize this will outrage someone somewhere. And yes, the operators have gotten a bit out of hand in the collections library. I put them all into a table, and the result wasn't pretty. (The problem isn't the operators, but their inconsistency.) There is a lot to like about the collections library, but it's not perfect. Perhaps it needs to get redone one more time, or perhaps Scaladoc needs to be smarter about giving a more user-centric view.

Do you have suggestions regarding inconsistency of operators? Given a long enough deprecation cycle we could do changes if they are necessary. One other thing I thought of was merging the Traversable and Iterable layers if collections, because this would get rid of about 20% of the supertraits of standard collection classes.
I think two of the things we need to do on the docs side is hide implementation details better in the ScalaDoc and put a link to the collections API overviewhttp://www.scala-lang.org/docu/files/collections-api/collections.htmlinto the scaladoc of each collection class.

Some of the other issues raised surely resonate with most of us: IDEs,
sbt/Maven, version incompatibilies. That will take blood, sweat, toil,
and tears to work through. Eventually it will all work, but you can't
blame a user to be frustrated about it.

I'm a little worried about further hiding implementation details in
the docs. Part of Coda's point was that all the hiding and such
promote cargo cult engineering. I'm not sure how much there is to be
done about it, but it definitely seems like trying to hide things like
type class implicits in the docs is only going to lead to arguments of
the form "Scala is so complex that they have to hide it from you!"

Maybe that's not what you meant, but it definitely seems worrisome. I
don't have any constructive ideas, unfortunately.

On Fri, Dec 2, 2011 at 11:08 AM, martin odersky <martin [dot] odersky [at] epfl [dot] ch> wrote:
> I think two of the things we need to do on the docs side is hide
> implementation details better in the ScalaDoc and put a link to the
> collections API overview
>
> http://www.scala-lang.org/docu/files/collections-api/collections.html
>
> into the scaladoc of each collection class.

I'm a little worried about further hiding implementation details in
the docs. Part of Coda's point was that all the hiding and such
promote cargo cult engineering. I'm not sure how much there is to be
done about it, but it definitely seems like trying to hide things like
type class implicits in the docs is only going to lead to arguments of
the form "Scala is so complex that they have to hide it from you!"

The point is that Scala's collections are fully extensible, whereas in many other languages they are not (that means the collections you can write there do not have the same status as the built in collections). To make extensions smooth we have to expose factory classes, builders, builder factories, and the like. I believe it is not at all cargo cult engineering if you program with collections without knowing the details. Engineers need to keep abstractions separate. Bridge builders need not know quantum mechanics for their work, even though it might be interesting for them to educate themselves in it. Collection users need not know the bits collections are composed from; the public user API is all they need. The main problem is that, besides in documentation, we cannot really distinguish user and creator APIs.
Cheers -- Martin

On 12/03/2011 07:07 PM, HamsterofDeath wrote:
> Am 02.12.2011 23:23, schrieb Tony Morris:
>>
>> Are you going to let wrong people affect the API of scala? It can only
>> get better from here!
>>
>> On Dec 3, 2011 7:55 AM, "Simon Ochsenreither"
>> > > wrote:
>>
>> I know, it sounds lame ... but can we get rid of /:, :\ and /:\?
>>
>> I'm sick and tired that the conclusion of “language experts” that
>> Scala sucks is based to 90% on those three method names.
>>
> tell them to use fold, foldLeft and foldRight if they don't want to use
> non letter methods.
>
> if i listened to conclusions like this, i would now be very, very
> confused because there are so many religions, and i am supposed to join
> any of them because they are all correct.
>

When I present Scala to my students I show them the operator-like methods but also the named methods. Then I say, “Personally I prefer the named methods because I find them easier to read in the code later. I’ll let you play around with both and decide for yourself.”

I find about 50% of the students will use the operator-like methods and 50% will use the named methods. That’s fine with me.

> -----Original Message-----
> From: scala-debate [at] googlegroups [dot] com [mailto:scala-
> debate [at] googlegroups [dot] com] On Behalf Of David Hall
>
> I'm a little worried about further hiding implementation details in
> the docs. Part of Coda's point was that all the hiding and such
> promote cargo cult engineering. I'm not sure how much there is to be
> done about it, but it definitely seems like trying to hide things like
> type class implicits in the docs is only going to lead to arguments of
> the form "Scala is so complex that they have to hide it from you!"

I agree that hiding things in the documentation, especially documentation
that is intended to be referential in nature, is not a good idea. On the
other hand making it clear where the extension points are would be good. I
can also understand hiding private API information for some suitable
definition of "private." On the other hand seeing all that weird looking
stuff in the scaladocs can also be motivational. It goes like this:

When I present Scala to my students I show them the operator-like methods but also the named methods. Then I say, “Personally I prefer the named methods because I find them easier to read in the code later. I’ll let you play around with both and decide for yourself.”

I find about 50% of the students will use the operator-like methods and 50% will use the named methods. That’s fine with me.

(I would dump /:\, though, as it serves little purpose. "fold" is clearer and faster to type (for most people), and /:\ suffers from which-goes-on-what-side ambiguity. (Especially given that people typically start with left folds to avoid stack overflow.))

I have no doubt that people who are interested in it will easily understand it.

But I'm really sick and tired of refuting the quick impressions from the same, sad piece of code written in 2008 for the next decade. It will never stop. Malicious people will always use this method against the language.

I would like to spend my time doing something useful for people interested and not waste my time by dubious "Java experts" writing pages of "gut feelings" of code they don't like in a language they never used.

The point that he makes that (to me, at least) resonates most strongly relates to the Scala community. I don't know how to fix it, but we really do need to fix it.

It must be possible to create a supportive and welcoming environment for people coming to the language. Right now, the Scala community is nothing like that.

Sadly, I must agree. However, I think it's a fairly small number of people who give this impression. I have posted many inane, newbie questions and received polite, thorough responses from people who I'm sure have better things to do with their time.So I'm going to call what you've pointed out the "FP" response, though it's not limited to FP, and certainly many hardcore FP-ers don't share this mindset. Because Scala is good at supporting different programming paradigms (FP, Actors, whatever), it can attract people who are locked into those mindsets; and if some of those people decide to move their attitude to the mailing lists, the result is pretty nasty.I can think of only two responses. I'm reluctant to even suggest the second, but think it must be said:1) Have Martin periodically post a message reminding everyone that Scala is a multiparadigm language, and asking for tolerance for other points of view.2) If necessary, banning people from a list. I don't like it, but if the atmosphere on the lists might cause Scala to fail, I think it's worth considering.Ken

On 03.12.2011 14:24, Peter C. Chapin wrote:
> definition of "private." On the other hand seeing all that weird looking
> stuff in the scaladocs can also be motivational. It goes like this:
>
> 1. What the $%&%$ is that?
> 2. Hmm.
> 3. Google/Books/Papers/Tutorials
> 4. Interesting.
> 5. Cool!

Actually that was sort of how it was for me when I first discovered
Scala, and was part of what attracted me to it. All that new weird,
"complex" stuff seemed like a magical forest waiting to be explored. It
brought the fun back in programming, and I hadn't quite felt that way
since I learned assembly in my late teens.

The point that he makes that (to me, at least) resonates most strongly relates to the Scala community. I don't know how to fix it, but we really do need to fix it.

It must be possible to create a supportive and welcoming environment for people coming to the language. Right now, the Scala community is nothing like that.

Sadly, I must agree. However, I think it's a fairly small number of people who give this impression. I have posted many inane, newbie questions and received polite, thorough responses from people who I'm sure have better things to do with their time.
So I'm going to call what you've pointed out the "FP" response, though it's not limited to FP, and certainly many hardcore FP-ers don't share this mindset. Because Scala is good at supporting different programming paradigms (FP, Actors, whatever), it can attract people who are locked into those mindsets; and if some of those people decide to move their attitude to the mailing lists, the result is pretty nasty.
I can think of only two responses. I'm reluctant to even suggest the second, but think it must be said:1) Have Martin periodically post a message reminding everyone that Scala is a multiparadigm language, and asking for tolerance for other points of view.
2) If necessary, banning people from a list. I don't like it, but if the atmosphere on the lists might cause Scala to fail, I think it's worth considering.Ken

The point that he makes that (to me, at least) resonates most strongly relates to the Scala community. I don't know how to fix it, but we really do need to fix it.

It must be possible to create a supportive and welcoming environment for people coming to the language. Right now, the Scala community is nothing like that.

Sadly, I must agree. However, I think it's a fairly small number of people who give this impression. I have posted many inane, newbie questions and received polite, thorough responses from people who I'm sure have better things to do with their time.
So I'm going to call what you've pointed out the "FP" response, though it's not limited to FP, and certainly many hardcore FP-ers don't share this mindset. Because Scala is good at supporting different programming paradigms (FP, Actors, whatever), it can attract people who are locked into those mindsets; and if some of those people decide to move their attitude to the mailing lists, the result is pretty nasty.
I can think of only two responses. I'm reluctant to even suggest the second, but think it must be said:1) Have Martin periodically post a message reminding everyone that Scala is a multiparadigm language, and asking for tolerance for other points of view.
2) If necessary, banning people from a list. I don't like it, but if the atmosphere on the lists might cause Scala to fail, I think it's worth considering.Ken