As species go, humans aren't renowned for their sense of smell. But an improved ability to suss out scents in our 200-million-year old ancestors may have laid the groundwork for the bulging brains of humans and all other mammals.

Virtual three-dimensional 'casts' of the fossilized skulls of animals that preceded the first true mammals suggest that brain areas involved in smell, or olfaction, catalysed brain growth in the evolutionary branch that gave rise to mammals.

With this foundation in place, later mammals could have siphoned off some of those resources for colour vision, echolocation and even, in the case of the platypus, the ability to sense electric currents. "The olfactory system was the thing that drove the expansion of the brain in the first place, and once you've got a big brain you can do all kinds of things with it," says Timothy Rowe, a palaeontologist at the University of Texas in Austin, who led the study, published online today in Science.

Brain boom

The evolutionary branch that spawned mammals took shape during the late Permian, about 260 million years ago. Known as cynodonts, these not-yet-mammals resembled reptiles, with small brains relative to their body size and puny olfactory bulbs -- the brain structures that pass sensory information about smells on to other parts of the brain -- Rowe says. "They had bad eyes, bad ears and a poor sense of smell, and from the structure of their brain it doesn't look like they were very coordinated."

What happened to the proto-mammalian brain next has mostly been speculation. Few well-preserved skulls documenting the transition exist, and real insight into brain anatomy requires the taking of 'endocasts' that capture an impression of the space the brain occupied in the skull. "To do anything means taking a rare skull and breaking it up. Palaeontologists aren't very fond of you doing this," says Glenn Northcutt, an evolutionary biologist at the Scripps Institution of Oceanography in San Diego, California.

Changes in the jaws of early cynodonts hinted that hearing might have driven brain expansion, Northcutt says. Alternatively, brain growth in mammals could have occurred across all regions by stretching out development so that the brain grows for a longer period, says evolutionary biologist Robert Barton at Durham University, UK.

To get a more concrete handle on how the brains of mammalian ancestors evolved, Rowe's team took high-powered X-ray images of two tiny fossil skulls unearthed in China and dating back to 175-200 million years ago. Using images taken from different angles, the authors melded these into three-dimensional computer tomography scans, analogous to brain endocasts.

One of the fossilized creatures, Morganucodon oehleri, had a puny head -- just 1.3 centimetres long -- but Rowe says it probably resembled animals such as opossums, with a fur coat and long tail. "If you were to look at it today you'd say: oh that's a mammal or that's almost a mammal. It would look very familiar to you," he says.

Compared with earlier cynodonts, Morganucodon boasted a bigger brain relative to its body size -- although still smaller than those of extant mammals. Brain areas involved in detecting and processing smells, which lie near the front of the skull, explain much of this growth, Rowe's team concludes. A region called the neocortex, which may have processed sensory information from the skin and hairs, also swelled in Morganucodon, as did the cerebellum, which coordinates movement.

The tiny skull of a second Chinese fossil, Hadrocodium wui, is even larger relative to its Lilliputian body and similar in terms of brain-body size ratio to some of today's mammals. Rowe's team recorded yet more growth in brain areas attuned to scent in this fossil, so Hadrocodium 's sense of smell may have been even keener than that of Morganucodon.

Common scents

The first true mammals may have gained even better senses of smell by developing additional nasal tissue to support the neurons that detect different odours and ferry them to the brain, Rowe says. Mammals have about 10 times as many odour-sensing genes as other vertebrates.

Rowe speculates that an improved sense of smell would have been useful toMorganucodon,Hadrocodium and their ilk because they would, by necessity, have become creatures of the night. "They were after insects and grubs and other things that were active at night," he says. "Dinosaurs were picking up on food sources active during the day and mammals took over the night shift." His lab is currently looking for fossil clues that early mammals were nocturnal.

Barton calls the new work "elegant and careful", and he agrees that it puts smell at the centre of the evolution of the mammalian brain. However, he also notes that our ancestors grew larger cerebellums -- essential for coordinating movement -- than did their predecessors.

Smells, after all, are useless if you can't act on them, Barton points out. "Ultimately, the function of the brain is to mediate adaptive behaviour, it's not to reflect on the mysteries of the Universe, much as we like to think that's what our brains evolved for," he says.

This article is reproduced with permission from the magazine Nature. The article was first published on May 19, 2011.

Scientific American is part of Springer Nature, which owns or has commercial relations with thousands of scientific publications (many of them can be found at www.springernature.com/us). Scientific American maintains a strict policy of editorial independence in reporting developments in science to our readers.