Spark submit hangs after writing from Cassandra to json

I have a driver program where I write read data in from Cassandra using spark, perform some operations, and then write out to JSON on S3. The program runs fine when I use Spark 1.6.1 and the spark-cassandra-connector 1.6.0-M1.

However, if I try to upgrade to Spark 2.0.1 (hadoop 2.7.1) and spark-cassandra-connector 2.0.0-M3, the program completes in the sense that all the expected files are written to S3, but the program never terminates.

I do run

sc.stop()

at the end of the program. I am also using Mesos 1.0.1. In both cases I use the default output committer.

Edit: Looking at the thread dump below, it seems like it could be waiting on: