Friday 13 November 2009 03.43 EST
First published on Friday 13 November 2009 03.43 EST

Earlier this year Glenn Beck, the US Fox News commentator, called President Barack Obama "a racist" with a "deep-seated hatred for white people and white culture". The subtext of the statement seemed to be that it is justified to be fearful and suspicious of people of another race if they hate and fear you. Or possibly it was just a more than usually sanctimonious form of racism. But for me it was also the spur to take a closer look at a book that charts the way American and European scientists have handled the debate about race, culture, intelligence and economic and political success.

That book is Stephen Jay Gould's The Mismeasure of Man, which seemed ground-breaking when it first appeared in 1981. It still seemed pretty good when Gould revised and expanded it in 1996, two years after two academic researchers published The Bell Curve, a book claiming to show that some hereditary lineages are innately less intelligent than others, leaving readers to draw the implication that money spent on educating them might be wasted. You can guess which lineages the authors might have included in this subset.

The Mismeasure of Man by Stephen Jay Gould Photograph: Public Domain

What Gould's book reminds us over and over again is that even very clever, generous and thoughtful people who are raised with a set of ingrained assumptions are likely to find evidence to support those assumptions.

Benjamin Franklin wanted a white America: he asked "Why increase the Sons of Africa, by planting them in America, when we have so fair an opportunity, by excluding all blacks and tawneys, of increasing the lovely white and red?" Thomas Jefferson thought that "the blacks … are inferior to the whites in endowment both of body and mind." Abraham Lincoln contemplated the physical differences between black and white and came out "in favour of having the superior position assigned to the white race".

The great 19th century scientists Cuvier, Humboldt, Lyell and Darwin all said things that betrayed an unquestioning belief in innate Caucasian superiority. Their successors set out to confirm this belief. Louis Agassiz, a great 19th century scientist now in the US Hall of Fame, thought social equality between black and white a "practical impossibility" and intermarriage "a perversion of every natural sentiment."

Some 19th century biologists argued that black people were the product of a separate creation, others that black people were inferior varieties of the same human species. A physician from Louisiana even argued in scientific papers that the people of Africa were "unable to take care of themselves" because of a disease of inadequate breathing "conjoined with a deficiency of cerebral matter in the cranium".

The idea that intellect had something to do with cranial capacity was – and to some people, still is – an attractive one, and generations of researchers tried to find new ways to measure brain size and shape, and match it with apparent intellectual performance. These experiments tended to prove that white people were cleverer than black people because they were bigger-brained.

In The Mismeasure of Man, Gould revealed that they could only prove this by massaging the results, cooking the data, and eliminating the unwelcome findings. One researcher found that German brains, on average, weighed 100 grams more than French brains. He was, of course, German. Measurements also produced inconsistencies: some Caucasian geniuses had very big brains, other intellectual giants had a quite modest cranial capacity.

So the anthropologists, anatomists and pioneer psychologists started looking for other things. They tried to grade the intellectual status of men, apes and women; of Nordic, Slavic and Mediterranean races; of long-headed and broad-headed peoples; they graded them according to the average distance between penis and navel, on the closeness of their eyes, on the lowness of their foreheads.

Then they began looking for ways to quantify the intellectual performance of different national and ethnic groups: and came up with bizarre results, which ought to have eliminated discrimination purely on the grounds of colour or race but somehow did not. In the early 20th century HH Goddard tried out his intelligence tests on new migrants and found, says Gould, that "83% of the Jews, 80% of the Hungarians, 79% of the Italians and 87% of the Russians were feeble-minded."

Robert Yerkes, another scientist still honoured among US researchers, tested military recruits and produced data that seemed to show that the mental age of the average white American was "about 13 years". Yerkes' tests suggested that the group whose intellects were below this average of 13 years included "37% of whites and 89% of negroes". This extra-low mental age did not disqualify black people from the Army because, said Yerkes, all officers seemed to agree "that the negro is a cheerful willing soldier, naturally subservient".

And so the whole, sorry, miserable story continues. These transparently silly and shameful "findings" were used to justify racial segregation in the American south, and to limit black youngsters' access to higher education. These limits, constraints and segregation laws continued well into the second half of the 20th century – well into Gould's lifetime, and mine.

This book should make any sensible person wary of attaching too much value to IQ tests (there's some glorious stuff on the quixotic allotment of IQ ratings) and should make anybody very suspicious of statements about "group IQ" or the presumption that some races are innately more clever than others. If we all got it so shockingly wrong 150 and 100 years ago, and even 50 years ago, then why would we have got it right now?

But there is another, deeper lesson in this book. The people who debased the science of humankind rubbed shoulders with the people who successfully shaped the rest of modern science, from Faraday to Einstein and Dirac, from Thomas Henry Huxley to Watson and Crick.

Scientists find it possible to be objective about the consequences that follow from the discovery of the speed of light in a vacuum, or the architecture of the double helix, or almost any subject except perhaps the human race. But when we look at ourselves, we see from a limited viewpoint.

"Know then thyself, presume not God to scan," said Alexander Pope, "The proper study of mankind is Man." Alas, when we contemplate ourselves, we can hardly claim to be objective.

Cover art: Seeing and Believing by Richard Panek

The International Year of Astronomy is drawing to a close, but the great adventure goes on. Next month, we take a look at Seeing And Believing: How the Telescope Opened our Eyes and Minds to the Heavens, Richard Panek's history of the instrument that launched the scientific revolution 400 years ago. The discussion starts on Friday 18 December