Although the granary is easy to build as far as technology and resources are concerned, it is not always useful. The reason for this is that on advanced levels of the game, the rate limiting factor in city growth is not the food or the doubling rate, but civil unrest. When the population grows at King difficulty level or above, the citizens get unhappy and unproductive very quickly. Thus, encouraging population growth just gets your cities to their point of unrest faster.

On higher levels of Civilization, then, the secret is not to build granaries to expand population. It is to build temples, marketplaces and harbors to keep the people happy, so that the population can keep expanding.