The master processes the split but when it calls ZKAssign.deleteNode it doesn't check the boolean that's returned. In this case it was false. So for the master the split was completed, but for the region server it's another story:

Since the znode isn't really deleted, it thinks the master just haven't got to process its region thus waits which leaves the region unavailable.

We need to just retry the delete master-side ASAP since the RS will wait 100ms between retries.

At the same time, it would be nice if ZKAssign.deleteNode always printed out the name of the region in its messages because it took me a while to see that the delete didn't take affect while looking at a grep.

Ted Yu
added a comment - 16/Nov/11 01:00 A loop was introduced which doesn't have sleep().
Should we exit the loop after certain number of attempts (certain amount of time)?
Also:
* @ throws KeeperException.NoNodeException if node does not exist
*/
public static boolean deleteNode(ZooKeeperWatcher zkw, String regionName,
EventType expectedState)
We should handle KeeperException.NoNodeException differently from other KeeperException's.

Jean-Daniel Cryans
added a comment - 16/Nov/11 01:05 It mirrors what's done RS-side, there's no limit there either. We could fix that too if we want.
The reason there's no sleep is because the RS is sleeping and we don't want to hit the same race another time.