Microsoft Azure Stack is an extension of Azure – bringing the agility and innovation of cloud computing to your on-premises environment and enabling the only hybrid cloud that allows you to build and deploy hybrid apps anywhere. We bring together the best of the edge and cloud to deliver Azure services anywhere in your environment.

HDInsight tools for IntelliJ & Eclipse June updates

HDInsight tools June release has been released! In this release, you can choose SBT as a build tool in addition to Maven when creating Spark projects in IntelliJ. With improvements to Spark job view and job graph in IntelliJ and Eclipse, more job info and statistics are now provided. You can also view job logs including driver stderr, stdout, and directory info easily in Spark job view.

Improved Spark job view and job graph in IntelliJ and Eclipse

The left pane of Spark job view shows all the Spark applications that ran in the cluster. Select one Spark job to view more details.

By hovering over the job graph, it will show the job run information. When clicking on job graph, it dives into the stage graph and shows the statistics of the job. You can also open Spark history UI or Yarn UI by clicking the respective link at the top of the Spark job view.

Improved Spark log view in IntelliJ and Eclipse

In the same Spark job view pane, click the Log tab to view the frequently used logs including driver stderr, driver stdout, and directory info, as shown below.

SBT build tool support when creating Spark project in IntelliJ

As shown below, you can now choose SBT as a build tool in addition to Maven when creating a new Spark project in IntelliJ.

Following the wizard to create a Spark project, after it is done, a new build.sbt file is generated which contains the build description for the project. Then you can author, submit, or debug the Spark job following the Spark job submission/debugging instructions.

How to install/update

HDInsight Eclipse plugin: Eclipse will prompt you for latest update if you have the plugin installed before, or you can get the latest bits by going to the Eclipse repository and searching “Azure Toolkit for Java”.

HDInsight IntelliJ plugin: IntelliJ will prompt you for latest update if you have the plugin installed before, or you can get the latest bits by going to the IntelliJ repository and searching “Azure Toolkit for IntelliJ”.

Feedback

We look forward to your comments and feedback. If there is any feature request, customer ask, or suggestion, please send us a note to hdivstool@microsoft.com. For bug submission, please open a new ticket using the template.