I'm currently the proud owner of an 8-node cluster that won't start up.

Yesterday we had a developer doing very high volume writes to our cluster via a Hadoop job that was reading an HDFS file and running six concurrent mappers on each of 8 nodes and using Hector to do the load and it sort of killed Cassandra. It was running 0.7.0 and actually killed three of the nodes with OutOfMemory errors before he realized something was awry and killed the job. He then tried to get rid of the keyspace by dropping it in the CLI and got the following error:

So he punted to me, and I decided to just try restarting the cluster in the hopes that it would sort itself out. The nodes that were still up died gracefully with the stop-server command, no kill -9s required. But when I tried to start the nodes again, they all failed with stack traces.