Jonathan -
I patched in your latest change that dropped the assertions and tried
to restart my cluster on my old data. 2 of 5 nodes still failed to
start, with different errors.
One dies with a generic EOFException during recovery:
INFO - Compacting
[/var/lib/cassandra/data/Analytics/movie-53-Data.db,/var/lib/cassandra/data/Analytics/movie-54-Data.db,/var/lib/cassandra/data/Analytics/movie-55-Data.db,/var/lib/cassandra/data/Analytics/movie-56-Data.db]
INFO - Completed flushing Memtable(country)@1074231936
ERROR - Exception encountered during startup.
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readUTF(DataInputStream.java:592)
at java.io.DataInputStream.readUTF(DataInputStream.java:547)
at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:222)
at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:320)
at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:58)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:91)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:167)
Exception encountered during startup.
The other dies with a more interesting:
INFO [main] 2009-10-16 21:42:17,082 Memtable.java (line 186) Flushing
Memtable(provider)@322343927
INFO [main] 2009-10-16 21:42:17,179 Memtable.java (line 220)
Completed flushing Memtable(provider)@322343927
ERROR [main] 2009-10-16 21:42:17,338 CassandraDaemon.java (line 185)
Exception encountered during startup.
java.lang.RuntimeException: Unable to load comparator class ''.
probably this means you have obsolete sstables lying around
at org.apache.cassandra.db.ColumnFamilySerializer.readComparator(ColumnFamilySerializer.java:108)
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:83)
at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:229)
at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:320)
at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:58)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:91)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:167)
Caused by: java.lang.ClassNotFoundException:
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.cassandra.db.ColumnFamilySerializer.readComparator(ColumnFamilySerializer.java:104)
... 6 more
There may be some unprintable garbage characters in the class name.
They show up in emacs but aren't visible in the system log file.
Would you expect the previous fatal error to have corrupted my data?
Edmond
On Fri, Oct 16, 2009 at 1:23 PM, Jonathan Ellis wrote:
> Don't know if you tried this yet, but Eric noticed a bug in my patch.
> There's a new one now, fixed.
>
> -Jonathan
>