SBT compiler

Although “SBT compiler” sounds like a completely new kind of Scala compiler, that is not really so. To see what SBT compiler is and how it fits into Scala compilation, let’s start with a “big picture”.

Here are the main tasks we need to perform during Scala project compilation:

We need to compile source code to bytecode (and to search for all kind of errors along the way). This task is always performed by Scalac (or Javac, for Java sources) no matter what tool is used to invoke it (IDE, SBT, Maven, Ant, Zinc or whatnot). So, Scalac (and Javac) is the actual “true” compiler, however it knows nothing about project structure, incremental compilation and so on. All it can do is to take a list of source files and produce a set of corresponding compiled classes.

We need to perform compilation incrementally (as opposed to compiling all sources all the time). Strictly speaking, this task is not required for project compilation (and it can even slow down a full project build). The whole point here is to save time during subsequent compilations. To do that, we need to track code changes and re-compile only affected files. And it is harder than it sounds — we need to monitor timestamps, store source-to-output mappings, check for public API changes, analyze direct and transitive dependencies (including ones between Scala and Java) and so on. There are not many tools capable of such a feat, and that is exactly what SBT “compiler” (as well as IDEA) does.

So, bytecode is still generated by Scalac (and Javac), project format is still IDEA’s, but incremental compilation is now handled by SBT instead of IDEA (so, we don’t use SBT project model, only SBT compiler, that acts as a wrapper around Scalac).

What’s the point of such an update? – Because there is a qualitative difference between how IDEA and SBT discover dependencies and process changes.

IDEA analysis is bytecode-based and language-agnostic. While it’s a very fine-grained, robust and time-proved method, there may be some rare unexpected glitches, specific to Scala code.

For example, let’s define a class C (in C.scala):
and then “add” a method name via an implicit conversion (in Main.scala):
After we compile and run the main method, we will see that the output is “foo”, and that’s what we expect it to be.

Then, let’s add a “real” method name to the class C:
After the next compilation, we will discover that the output is … still “foo”, and that is is not what we expected… Why so? – Because, in terms of the bytecode, there’s no need to recompile Main class yet.

There are many additional cases (for example, related to named- and default parameters) which demonstrate, that bytecode analysis is not enough for proper incremental compilation of Scala code.

That is where SBT compiler comes to rescue, because it relies on source-based analysis of compiled code (to do that, SBT intercepts internal Scala compiler data), so it can take account of Scala-specific details to perform incremental compilation correctly (though SBT analysis of Java classes is still bytecode-based).

As a bonus, because SBT uses a so-called compiler interface layer to communicate with the Scala compiler, we can now run multiple versions of in-process Scala compilers in the same JVM. That’s how we can:

reap the benefit of JIT compilation (either during compilation of multiple modules or when using a compile server),

avoid creation of temporary files (with compiler arguments),

avoid compiler output parsing,

make compiler JVM parameters (in Scala facet) obsolete.

It’s worth noting, that there is a one side-effect of using SBT compiler, resulting in how Java files are handled in Scala projects (ones with a least one Scala facet). Because we need to maintain a coherent model of dependencies, compilation of all Java files, event in “pure” Java modules, is routed through SBT. For most use cases that is perfectly OK, unless you need to use a non-standard Java compiler (by the way, SBT supports in-process Java compiler as well).

New compile server

Scala compiler is a quite complex piece of software, so it takes time to load and initialize it. When you compile a project with hundreds (or thousands) of files, this time is negligible comparing to the time needed to compile the code. However, when incremental compilation is used, and all you need is to re-compile a few files, it may take longer to warm-up a Scala compiler than to actually compile those files.

One solution to this problem is to run Scala compiler in-process to avoid compiler re-instantiation when compiling subsequent modules. This, however, still cannot liberate us form a first compiler initialization. To speedup compilation further, we need to use a pre-initialized compiler instance, and that is exactly what compile server does.

Originally, Scala compile server was implemented via FSC (Fast Scala Compiler). While it works well enough, it has several limitations:

FSC instance is bound to a single compiler version, so compile server settings were a part of project configuration and FSC instances are not shared between projects.

Because FSC reuses symbol table between runs, it sometimes might produce various errors during compilation.

Now, because of our newfound ability to run multiple versions of Scala compiler in a single JVM, we can do better than that. New compile server is implemented using Nailgun (basically, it’s just a JVM process that can be accessed via a TCP/IP connection), so:

It is now a matter of personal preference whether or not to use a compile server. Compile server configuration is moved from project settings to application settings.

Compile server is now application-wide – it is reused for multiple compiler versions and shared between all projects (so we can save a lot of RAM).

Compile server configuration is now located in Settings / Scala:

It’s recommended to use the following JVM parameters for the Scala compile server:

-Xss1m – to ensure that there will be enough stack space for recursive code in Scala compiler.

If compile server is not used, it’s still recommended to use the same parameters (except, maybe, -server) in Project Settings / Compiler / Additional compiler process VM options.

How all this is related to Zinc

Zinc is a stand-alone launcher for SBT incremental compiler (and Nailgun server). It was created, primarily, to be used in command line tools (like Scala Maven Plugin).

Because both Zinc and the new compilation subsystem use the same tools and techniques, they are essentially equivalent. There is not much sense in using a launcher from another launcher, so we run both SBT compiler and Nailgun directly, rather than via Zinc.

Besides, we need much more data from the compiler (to display compilation messages, report progress, compile UI forms, run annotation processors, etc.) which is not available from Zinc output. Zinc’s Nailgun server also cannot be used by IDEA because its process is started without IDEA-specific classes (communication protocol) in JVM classpath.

As Zinc cannot build a project by itself, it must be invoked from another tool (like Maven), which stores all incremental compilation data using its own scheme, that differs from IDEA’s one. So, currently we cannot unite IDEA’s and Zinc/Maven’s incremental compilation even though they are very much alike.

Summary

Let’s sum-up all the advantages of the new Scala compilation subsystem:

Less load on IDEA process.

External project build.

Background compilation.

Source-based incremental compilation.

In-process Scala and Java compilers.

Simplified project configuration.

Application-wide compile server.

There must be a catch in it somewhere, you know… And here it is – because the new compilation subsystem is written completely from scratch, some bugs are simply inevitable. Please, report them so we can fix them as soon as possible.

Keep in mind, that the previous compilation subsystem is still available, and can be always turned on by clearing “Use external build” checkbox in Project Settings / Compiler page:

You may also check there whether the external build is enabled in your existing projects.

For those who want to know even more details, here is source code for all the tools:

123 Responses to A new way to compile

FSC used to eat tons of memory during often builds on several projects.
If I understand correctly Nailgun should reuse some memory in single process?
At least when the compilers for different modules are the same.

I was unable to get existing projects to use it though, even if I removed the Scala facet and re-added it. The “Run compile server (in external build mode)” is checked. For existing projects under Project / Compiler / Scala it still has the old settings (Joint Compilation / Project FSC), instead of “Nothing to show here in external build mode”.

Workaround is to remove .idea and reimport the project from the pom.xml.

I may be a little confused here regarding the external build and the Scala compile server. The above error occurs when external-build, run-compile-server and automake are selected. How do you turn on automatic background compile when “Run compile server” is checked?

I don’t know generate income appeared here, however idea this kind of publish was excellent. I really don’t understand who seem to you are and surely you will some sort of well-known blogger while you are certainly not already. Cheers!

I really appreciate your ongoing work on improving the Scala support in IDEA. However, in this case I regret the recent development. IDEA had a great support for fsc (fast scala compiler) and it worked very well. The full and incremental compilation was fast and usually without recompilation issues.

Now you introduce a new compiler system and advertise it with “lower IDEA memory consumption”, “reduce load on IDEA’s code model”, and “seedup compilation” and I wonder what was your reference for that: the standard Scala compiler or fsc?

When I switched from fsc to the new external build server I noticed a dramatic regression regarding compilation time for my projects. The time for incremental compilations (after changing one code line) rose from 8s (fsc) to incredible 21s (new compile server). And even the compile time for a complete rebuild was with fsc 20s faster than with the new compile system. I posted my results on http://devnet.jetbrains.com/message/5475005#5491387. And regression got confirmed by http://devnet.jetbrains.com/message/5474953#5474953 and is still present.

As long as IDEA offered the option to switch back to fsc I didn’t care about this regression, but today after updating IDEA I regretfully noticed that this option has gone and we are forced to use the 2-3 time slower external compiler now. That’s really sad.

So what about your claims?
– “lower IDEA memory consumption” -> might be, but is definitely not significant (at least I don’t see it) regarding fsc
– “reduce load on IDEA’s code model” -> I don’t see a difference. I could always work with IDEA while fsc was compiling.
– “speedup compilation” -> That’s only true if you compare the new compiler regarding the standard scala compile (that nobody uses) but not fsc. And it’s probably true if you run the new compiler in parallel on multi module projects. But what’s the major benefit of that? The usual developer compiles his/her project around 100 times or more a day and does not work in multiple modules in parallel. He/she only wants a fast incremental compilation of the module he/she currently works on. And that’s exactly where the new compiler fails.

I’m sure you had the best intentions when you wrote the new compiler and could prove the benefits for some (more theoretical) test cases. It’s a pity that those cases did not include the most common ones of daily use.

For the future I hope that you manage to improve the compiler performance or bring back the old fsc integration again. However, IDEA is still the most advanced Scala IDE. I just don’t know if it’s worth to upgrade to IDEA 13.

from certain time. my intellijIdea does not work well with the “use external build” and i find “use external build” is a default select option.but from last friday.my svn resposity code is not working well will this option. when this option is selected .the compiler will compile nothing. and idea will start build artifact immediately.so ,what ‘s wrong with my config ?

On line coaching pertaining to school diplomas has a tendency to create
exceptional outcomes regarding a number of factors. The availability of 4GB capacity in both the SD and SDHC
families has caused much compatibility confusion with users since each has a slightly different communication protocol.
The Pandora 3DS Card is highly compatible with the latestcommercial ROM’s.

Do you have a spam problem on this blog; I also am a blogger,
and I was wanting to know your situation; many of us have created some nice
practices and we are looking to exchange strategies with others, be sure to shoot me an email if interested.

I have not checked in here for some time because I thought it was getting boring, but the last few posts are great quality so I guess I will add you back to my everyday bloglist. You deserve it my friend

The similar applies to this game as well and as a result most
of the participants are often on the lookout for the right hack and tool selections to increase their strength and also appreciably increase the
possibility of winning.

10:39:48 AM Scala compile server: java.net.BindException: Address already in use: JVM_Bind
at java.net.DualStackPlainSocketImpl.bind0(Native Method)
at java.net.DualStackPlainSocketImpl.socketBind(DualStackPlainSocketImpl.java:96)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:175)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.(ServerSocket.java:237)
at com.martiansoftware.nailgun.NGServer.run(NGServer.java:413)
at java.lang.Thread.run(Thread.java:722)

Simply desire to say your article is as amazing. The clearness on your put up is just great and that i can assume you are an expert in this subject. Fine along with your permission allow me to clutch your feed to stay up to date with forthcoming post. Thanks a million and please carry on the rewarding work.

Ordinarily I would not learn document for blogs, however would wish to express that this specific write-up quite required my family to take a peek with and practice it! The way of writing continues to be shocked me. Thanks, quite excellent article.. Agen Bandar Bola Judi Taruhan Online Terpercaya Terbesar

Definitely consider that that you stated. Your favorite reason seemed to be at the net the simplest thing to understand of. I say to you, I certainly get annoyed even as other folks think about concerns that they plainly do not recognize about. You managed to hit the nail upon the top as neatly as defined out the entire thing with no need side effect , other folks could take a signal. Will likely be back to get more. Thanks

Fantastic website! I located it while browsing on Yahoo News. Do you’ve any suggestions on ways to get listed in Yahoo News? I’ve been attempting for a while but I under no circumstances look to acquire there! A lot of thanks

Thanks a lot for sharing this with all of us you really realize what you’re
speaking approximately! Bookmarked. Kindly additionally discuss with my site =).
We may have a hyperlink trade arrangement between us

I do enjoy the manner in which you have framed this specific challenge and it really does provide me personally a lot of fodder for consideration. Nonetheless, through what I have observed, I only trust when the actual opinions pack on that men and women stay on point and not embark on a soap box involving the news of the day. Yet, thank you for this fantastic point and while I can not really agree with this in totality, I regard the perspective.

I want to blog about issues I don’t want associating with my ‘work’ self. How do I choose a blog host that has a good anonymity record, in other words, I don’t want it to be easy for people to match up my IP address or access my personal details.. Also, having set the blog up, how do you encourage high volume visitors?. I’ve only used LJ before and I want to try something a bit different, but anonymity is the biggest issue for me. Would welcome suggestions of good free blog hosts..

Heya excellent blog! Does running a blog similar to this take a lot
of work? I have very little understanding of computer
programming however I was hoping to start my own blog soon. Anyway,
if you have any recommendations or tips for new blog owners please share.
I understand this is off subject however I simply had to ask.

Thank you for any other fantastic article. The place else
may just anyone get that kind of information in such an ideal means of writing?
I have a presentation next week, and I’m on the search for
such information.

Hello there! This artidle couldn’t be written much
better! Going through this article reminds me of my previous roommate!
He constantly kept talking about this. I most certainly will send this article to him.
Pretty sure he’s going to have a geat read. Thank you for sharing!

Having read this I believed it was really enlightening.
I appreciate you spending some time and effort to put this informative article together.
I once again find myself personally spending a lot of time
both reading and leaving comments. But so what, it was still worthwhile!

Thank you for any other fantastic article. The place else
may just anyone get that kind of information in such an ideal means of writing?
if you have any recommendations or tips for new blog owners please share.
I understand this is off subject however I simply had to ask.

Our high class companions are as elegant as required and rather wholesome looking. Well spoken and elite in their tastes, the elite escorts are the perfect way to spoil yourself totally while you are in the CBD. http://asiabetkingbola.com/To really experience what it means to be pampered by a high class Hong Kong escort masseur, call one of our mosseur call girls. to your 5 star hotel and watch the events unfold as your deepest desires and wildest massage fantasies

We are specialists, the most troublesome calling on the planet, yet we have no insurance we get the most extreme training, however we have no barrier, We have no weapons, nor a legitimate assurance. Our preparation is the heaviest preparing on the planet, however we are helpless.