'Gerald Crabtree, a geneticist at Stanford university, claims that we are stupider than our hunter-gatherer ancestors'. Photograph: Philipp Kammerer/Alamy

What is it with geneticists and impending doom? Some of greatest evolutionary biologists of the last century were preoccupied with the idea that civilisation was ruining the human species and now another version of the idea has surfaced. Gerald Crabtree, a geneticist at Stanford University, claims that we are all stupider than our hunter-gatherer ancestors. They operated, he said, under much greater evolutionary pressures than we do, so that stupid people were eliminated from the gene pool.

Before going on to the ancestral roots of this idea, it's worth noticing one objection to his theory that fails comprehensively. If the hunter-gatherers were so smart, how come they are almost extinct? Over most of their range they seem to have been hunted to death (often literally) by more settled farmers. All that are left are a few scattered populations in places no one else wants to live, like the rainforest and the Kalahari desert.

This is true, but it is false to suppose that evolution must always select for intelligence. Domesticated animals have generally smaller brains than their wild ancestors precisely because they no longer need to make so many decisions. Yet they are also far more numerous and successful. All evolution cares about is how many of your grandchildren survive. It's not picky about how and why this happens.

The idea that civilised man is a degenerate and self-domesticated variation on the wild type is partly a cultural trope, a result of the anxieties of industrialised life. You'll find it in some of the most influential works of pre-first-world-war fiction: Jack London, for example. It's taken to a different extreme in EM Forster's novella, The Machine Stops.

But in the 1930s, it was one of the drivers of one of the most influential books in the history of biology: RA Fisher's The Genetical Theory of Natural Selection. This was central to the unification of genetics with evolutionary theory. Its conclusions lie at the foundations of modern biology. But if you actually read the original, you will find that the last five chapters – nearly half the book and, I think, the half the author thought most urgent and important – elaborate the idea that civilisation is dreadfully threatened by the way the lower classes outbreed the aristocracy.

Fisher took the idea of an aristocracy quite straight. Writing in the 1930s, he felt that even tentative moves towards greater equality, such as universal education, had diminished the purifying force of social selection. He was confident that in the 18th and 19th century people got rich by developing more interesting virtues and that the social promotion of the earlier period was predominantly conditioned by the qualities of active attention to business, enterprise combined with prudence, good repute and credit with neighbours, and intelligent adaptation to the changing conditions of trade; as well as the other more special gifts and aptitudes which enable such men to consolidate and enlarge their businesses.

There is, at least, a presumption that the thousands of relatively able children from the poorest homes who are now drafted into clerical and teaching occupations are more characterised by intellectual superiority, and less by enterprise, initiative, and responsible judgment, than were those who made equal social progress 100 years ago.

If Fisher thought that universal education made life too easy, his great successor WD ("Bill") Hamilton, thought we were threatened by spectacles, and still more by Caesarean sections: "I predict that in two generations the damage being done to the human genome by the ante-and postnatal life-saving efforts of modern medicine will be obvious to all."

Although he was heaped with honours after his death, and the subject of a truly lovely eulogy by Richard Dawkins, his embrace of euthanasia and eugenics as the only escape from genetic and personal catastrophe was so enthusiastic that the publication of the second part of his collected papers was delayed for years by embarrassment.

But worrying about intelligence, as Crabtree does, adds a new twist on this theme. It is especially odd in that the one thing we know about measurable intelligence is that it has risen steadily through the past centuries, something known as the "Flynn effect", after its discoverer, the New Zealand psychologist James Flynn.

And it is Flynn who has the most useful and interesting take on this. In his book "What Is Intelligence?", he takes seriously the idea that IQ tests measure something real. When the tests show that, for instance, American black men are 15 IQ points lower than their white contemporaries, this shows genuine cognitive differences and disability. But, he goes on to argue, these are not innate. There are clear and identifiable social factors which produce these kinds of group difference.

What is more, the kind of reasoning that an IQ test measures is quite finely adapted to the modern, technological world in which most people who take one have grown up. It rests on assumptions that simply don't hold in different societies. Flynn illustrates this with studies conducted on Siberian hunter-gatherers in the 1930s. Their answers, apparently stupid, were highly acute once you realised that they did not trust the questioners at all. In their society, suspicion was not in the least bit stupid as a general rule, no matter how much it might fail to provide useful answers in this particular instance.

It well may be that contemporary adults are hopeless stupid and inadequate by the standards of hunter-gatherer life and that adult hunter-gatherers would die quickly in a modern city, too. But neither of these tests are measures of innate intelligence, as Flynn makes clear. The real test would be to take babies from each society and bring them up in the other. And if you did that, you'd realise that the real selective pressures have nothing to do with intelligence, but with resistance to disease, and parasites, and hunger.