On transient config server failure, mongod should not abort

Details

Description

Certain failure modes of the SyncClusterConnection to the config server are safe failures (we are sure nothing has been written) - it is incorrect to fail by aborting mongod here since our state is fully known (the migration failed). Rollback and continue instead.

Original description:

If the config server is offline for a short amount of time during the critical section causing a write error but we're able to reconnect later, we should try to A) verify the write failure on all servers and B) locally rollback the new version, aborting the migration. Currently we fail hard, and rely on the mongod restart for the config reload.

Note this is different from the case when the config servers go down and stay down, since in that case we are unable to read the state of the metadata and so have to terminate.

One member (member1) in a set crashed, because of a server crash. This server also runs 1 of 3 config servers. After that another member (member2) crashed because it couldn't reach the crashed server? This is the message on member2:

Tue Sep 25 13:34:56 [conn538] DBClientCursor::init call() failed

Tue Sep 25 13:34:56 [conn538] scoped connection to config1:27019,config2:27019,config3:27019 not being returned to the pool

In this case, a config server went down in the middle of a moveChunk command, and the remaining config servers went read only. The moveChunk command attempted to write the new config with applyOps, read the old config, got the old version, and then aborted.