HAD-355: Blacklisted nodes are now added back into the cluster within a default interval of 2 hours. Earlier the blacklisting was permanent

SPAR-187: Increase spark's metastore timeout to 5 mins. Especially if the cluster is in a VPC, it takes quite a bit of time for a single metastore call to complete.

SPAR-174: Allow JsonSerde to work with SparkSQL.

Cluster usage report minor fixes.

SCHED-54 Fixed a bug when concurrency is more than 1

HADOOP

Qubole query id is now available in Hadoop jobs (via property qubole.command.id in job.xml)

Hadoop Jobs launched by Shell Commands have their parent/child information in job.xml

SPARK

SPAR-198: Container log links work from Spark Web UI in yarn-client mode. The hadoop user is picked up in SparkContext itself.

SPAR-197: Prepend local: to all jars in /usr/lib/spark/lib. This allows yarn-cluster mode to also process jars correctly.

SPAR-174: Configure /usr/lib/spark/lib as the standard jar destination for spark. Any jar in this dir is available to both driver and executor. We also use this mechanism to make hive_contrib.jar (JsonSerde) accessible to SparkSQL.