We are no longer monitoring this channel, please join Slack! https://join.slack.com/t/atomixio/shared_invite/enQtNDgzNjA5MjMyMDUxLTVmMThjZDcxZDE3ZmU4ZGYwZTc2MGJiYjVjMjFkOWMyNmVjYTc5YjExYTZiOWFjODlkYmE2MjNjYzZhNjU2MjY

I improved the fuzz test quite a bit to introduce a lot more issues - random large snapshots, random large commands, more clients, more commands, and adding/removing nodes from the cluster in addition to shutting them down. It’s doing a lot of work now.

I also set up some basic consistency tests. Copycat's incremental compaction algorithm actually makes it difficult to do assertions on the state of the state machine or the log at a specific point in time. Severs are able to compact their logs independently, and not all servers will see all commands, especially in these tests. So it's entirely possible for a follower's state machine to be inconsistent with the leader's state machine at a given point in logical time, but it will become consistent before servicing linearizable/sequential reads.

So, I had to do client side checks according to those consistency models. That's where they matter anyways. This is more or less what Jepsen does. Mine is just a more hacky version that covers Copycat's specific consistency models.

The tests seem to be doing pretty well now. I'll be running them for a while and then will prepare a release.