Description

However zk3 fails, restarts, and rejoins the quorum as the new leader (was the old leader, still the leader after re-election)

The existing two followers, zk1 and zk2 rejoin the new quorum again as followers of zk3.

zk1 then fails, the datadirectory is deleted (so it has no state whatsoever) and restarted. However zk1 can never rejoin the quorum (even after an hour). During this time zk2 and zk3 are serving properly.

Later all three servers are later restarted and properly form a functional quourm.

Here are some interesting log snippets. Nothing else of interest was seen in the logs during this time:

zk3. This is where it becomes the leader after failing initially (as the leader). Notice the "round" is ahead of zk1 and zk2: