So, the new RS3 shard appears in the list of shards, but not in the list of "how many chunks does each shard have". I would have expected it to appear in that list with a count of 0 for all sharded collections.

2 Answers
2

It will start to have chunks moved over to it, yes, in fact it will be the default target for every chunk move for the foreseeable future (basic selection is to move from shard with most to the shard with least chunks). Each shard primary can only take part in a single migration at a time, so with that many chunks to move it is going to take some time, especially if the other two are busy.

I have seen cases where people have turned off the balancer and forgot about it. Given that your other 2 shards are balanced pretty well, I don't think that is the case here, but just in case....

You can check on the status of the balancer by connecting to the mongos and then doing the following:

use config;
db.settings.find( { _id : "balancer" } )

Make sure that "stopped" is not set to true.

To see what is holding the lock, and hence balancing at that time:

use config;
db.locks.find({ _id : "balancer" });

Finally, to check what the balancer is actually doing, look at the mongos log on that machine. The balancer outputs rows to the log prefixed by [Balancer]. You can also look for migration messages in the logs of the primary mongod instances in the logs.

EDIT: This was likely caused by SERVER-7003 - a bug found in 2.2.0 post release. If there are deletes in the range (chunk) being migrated from the source shard, it can sometimes cause this sort of paralysis where all chunk migrations are aborted and the target shard appears to always be taking part in a migration, when in fact it is not.

Since this has been fixed in 2.2.1 an upgrade is the recommended path to resolve the issue. Though it can be resolved by restarts and/or when the bad state on the target shard resolves itself, as seems to be the case in the comments below.