Headhunters: The Search for a Science of the MindBy Ben Shephard (The Bodley Head 323pp £25)

Charles Myers recording the sacred song of the Malu ceremonies, 1898

Summing up the current intellectual situation, Ben Shephard writes: 'On the face of it we now live in a completely new world. All the old gods are dead - neither nationalism, Marxism, psychoanalysis nor Christianity any longer provides philosophical ballast. Instead, the modern intellectual landscape is dominated by two phenomena, Neo-Darwinian genetics and modern neuroscience - just as it was in the 1890s.'

As Shephard shows in this refreshingly sceptical mix of biography and intellectual history, the present intellectual climate is not as unprecedented as some would like us to think. The belief that a synthesis of Darwinism and neuroscience would revolutionise understanding of human behaviour was pervasive in the late 19th and early 20th centuries. Linking evolution with neurology, it was believed, would produce a new science of the mind, which would in turn transform our understanding of ethics, politics and the human species itself.

Now, at the start of the 21st century, Christianity may still be retreating in most Western countries (though the opposite is the case in China, Russia and much of the developing world), while Marxism and psychoanalysis may have faded from view, but the idea that we are on the brink of a scientific revelation regarding the nature of the human mind that will transform the way we think of ourselves is as strong as it has ever been. Yet any suggestion that the human sciences can be progressive disciplines like physics remains as problematic as it was a century ago, and the neo-Darwinian theories that proliferate at the present time will surely prove to be as misguided as those that flourished in late Victorian times.

Headhunters is a mind-opening exploration of the lives, careers and ideas of four leading figures who believed themselves pioneers in a new branch of knowledge: the anthropologist and psychiatrist William Rivers (1864-1922), Rivers's pupil the physician and psychologist Charles Myers (1873-1946), the anatomist Grafton Elliot Smith (1871-1937) and the social psychologist William McDougall (1871-1938). The origins of this arrestingly original and beautifully written book are themselves an interesting story. They go back to 1963, when as a teenager Shephard read Siegfried Sassoon's Sherston's Progress, in which Rivers appears as the doctor who treated Sassoon for shell shock. Reinforced by studies of the traumas experienced by servicemen as a result of the Vietnam War, Shephard's concern with the psychological aftermath of warfare led to an interest in military psychiatry, which eventually resulted in his writing War of Nerves (2000). All of the thinkers examined here, Shephard discovered, were involved in one way or another in the treatment of psychologically damaged soldiers. Rivers's pupil Myers coined the term 'shell shock' when serving in the army in France; McDougall - another pupil of Rivers - had written an account of the patients whom he had treated back in England; while Elliot Smith had published a wartime polemic on shell shock.

Shephard writes that he has 'tried to give some idea of how science actually works: the passions, the irrational flashes, the moments of insight - the big ideas that work and the big ideas that are plain wrong'. He does this, in part, by giving us a flavour of the personalities of his four protagonists. We find Rivers telling Sassoon - with whom he seems to have had an unspoken homosexual affinity - that Sassoon didn't have shell shock, but instead an 'anti-war complex', at which point both of them laughed. We hear of Myers, who had refused to give evidence at an inquiry on 'shell shock' in 1921 because he feared he would be embroiled in public controversy, returning to the subject only in 1940, when he declared the term he had himself invented to be 'singularly ill-chosen' and advocated harsh measures in order to curb the 'contagiousness' of the complaint. We learn of the gradual destruction of Elliot Smith's reputation by his refusal to accept the authenticity of the Australopithecus fossil (which pointed to the African origins of modern humans), his endorsement of Piltdown Man and the falsification of his theories of cultural diffusion by radiocarbon technology. In a particularly fascinating discussion, Shephard shows how McDougall's involvement with psychical research helped demolish his career: attacked in Britain by academic psychologists who feared the subject would undermine the scientific respectability of their discipline, McDougall was again attacked when he moved to America by spiritualists who thought his approach too sceptical and drily scientific.

Absorbingly readable as a study of four intertwined intellectual biographies, Headhunters is also compellingly plausible as a critique of the Whiggish history of science, which Shephard well describes as 'a narrative of heroic progress towards the present'. Nowhere is this edifying story more unrealistic than in the matter of race. Rivers, Myers and McDougall were members of the celebrated anthropological expedition to the Torres Strait Islands in 1898, and in 1901 an account of the expedition was published by its organiser, Alfred Haddon, under the title Head-hunters: Black, White and Brown. The purpose of the expedition was to study 'the primitive mind', using techniques that among other things involved measuring reactions to pain.

No one among the ship's scientists doubted for a moment the profound gulf between themselves and the human beings they were studying. 'All the great men of late-Victorian science firmly believed in a hierarchy of races,' Shephard writes with admirable forthrightness, 'none more strongly than Darwin, whatever his modern whitewashers would have us believe.' Darwin's populariser and 'bulldog', T H Huxley, developed a classification of racial types based on skin colour, hair colour and texture, eye colour, skull shape and body stature. Anthropology was seen as the study of 'lower races' - a view illustrated in the work of Huxley's grandson Julian Huxley, who wrote in 1931 of there being 'a certain amount of evidence that the negro is an earlier product of human evolution than the Mongolian or the European, and as such might be expected to have advanced less, both in body and mind'. In 1935, with the Nazi threat becoming clear, Huxley distanced himself from these views, pronouncing in 1935 that the concept of race was 'hardly definable in scientific terms'. In the few years that transpired between Huxley's two pronouncements, nothing had happened in science to change his mind. It was the world that had changed.

The repudiation of 'race' as a scientific category, which became more or less complete in the years after the Second World War, did not come about because scientists had recognised in the course of their inquiries that the concept was illegitimate. The idea was abandoned as a result of the military defeat of Nazism and the disclosure of what the Nazis had done in applying it. If Nazism had triumphed, or been allowed to prevail in Europe and expand in the East as part of a shameful peace of the kind advocated by appeasers, racial pseudo-science might well have continued to be taught in universities to this day. The idea of race as a key element in human evolution seems now to be making a comeback, and the belief that the human sciences can be progressive disciplines is being promoted through the revival of smelly old fallacies. Intellectual life is once again dominated by a rash of faux-Darwinian metaphors masquerading as revolutionary new paradigms - just as it was at the end of the 19th century.