Apache Spark continues to be one of the most prominent names in the world of Big Data technologies. Spark adds lots of features to the basic Scala language, but the 2 most prominent ones are RDD’s (Resilient Distributed Dataset) and the Spark Shell (REPL).

If you have ever tried programming in Scala to run on Apache Spark, you would likely get to a point that you would say to yourself “the shell is a nice tool for debugging , but i need more than this if i want to be more productive.” Modern IDE’s provide a large variety of tools to support enhanced productivity, tools like syntax highlighting, auto completion, debugging, and so on.

IBM BigInsights 4.1, IBM distribution for Hadoop, supports Apache Spark 1.4.1 out of the box. To work with a Spark cluster configured using BigInsights 4.1, you simply configure the same version of Spark on your desktop machine where IntelliJ is installed.