Spark Optimization

We are currently working on POC based on Spark and Scala.we have to read 18million records from parquet file and perform the 25 user defined aggregation based on grouping keys.we have used spark high level Dataframe API for the aggregation. On cluster of two node we could finish end to end job ((Read+Aggregation+Write))in 2 min

Cluster Information:Number of Node:2Total Core:28CoreTotal RAM:128GB

Tuning Parameter:spark.serializer org.apache.spark.serializer.KryoSerializerspark.default.parallelism 24spark.sql.shuffle.partitions 24spark.executor.extraJavaOptions -XX:+UseG1GCspark.speculation truespark.executor.memory 16Gspark.driver.memory 8Gspark.sql.codegen truespark.sql.inMemoryColumnarStorage.batchSize 100000spark.locality.wait 1sspark.ui.showConsoleProgress falsespark.io.compression.codec org.apache.spark.io.SnappyCompressionCodecPlease let us know, If you have any ideas/tuning parameter that we can use to finish the job in less than one min.