hi, I am playing again with newest version as I see many things have changed. I am a bit lost when testing adding new nodes and removing nodes from cluster. I see that atomix requires initial bootstrap nodes to form cluster but I cannot add new one later. I cannot see in documentation how to do that. Previously there was #join() method on Atomix instance, and now it is gone :(

I get error like this

12:23:31.292 [raft-server-core-partition-1] WARN MDC[] io.atomix.protocols.raft.impl.DefaultRaftServer - RaftServer{core-partition-1} - Failed to startserver!
Exceptionin thread "Thread-263" java.util.concurrent.CompletionException: java.lang.IllegalStateException: not a member of the cluster
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:769)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at io.atomix.protocols.raft.cluster.impl.RaftClusterContext.lambda$join$10(RaftClusterContext.java:413)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: not a member of the cluster
... 11 more
12:23:33.908 [raft-server-coordination-partition-4] WARN MDC[] io.atomix.protocols.raft.roles.FollowerRole - RaftServer{coordination-partition-4}{role=FOLLOWER} - io.atomix.messaging.MessagingException$NoRemoteHandler: No remote message handler registered for this message

yes I am building from master, as I am not aware of any nightly build that was published with 2.1.x version. I see now you are still modifying structure of the project and several modules were removed but were referenced from my local repo.

where bootstrapNodes consists of all primary nodes (3 nodes) + 2 new. Those 3 primary machine are running and are active when I try add those new one. code above is run from separate thread so #join() is not blocking progress of other Atomix nodes. I guess it is all right to have 3 machines with different configuration than for 2 joining.

@kuujo getting back to this after about a year. in the older world it was possible to use just copycat to have a replicated-state-machine and user could keep any arbitrary state in there . in current code, is it ok for user to use atomix-raft standalone the way I could use copycat before or is it necessary to use atomix-cluster and other things on top of raft? I'm looking to just use the raft implementation to construct replicated state machine in my application and I can't see StateMachine anymore.

@johnou but if you have time I can send you link to sample repo. I was planning to present atomic on minor conf next year but it is really hard to make a progress with new version and about old is waist of time.