It's a waxy layer that covers their bodies and is the source of the complex aromas that ants use to communicate. These odorant blends act like biochemical uniforms, identifying individual ants by caste, colony, and species. In so doing it helps regulate the ants' behavior, allowing them to navigate the sophisticated social systems that have made ants one of the most successful families of animals on Earth.

"Ants see their world through their nose, their antennae. If you're an ant, you view others by their smell and others view you by your smell," said Laurence Zwiebel, Cornelius Vanderbilt Professor of Biological Sciences, who is studying the molecular basis of ant olfaction.

For some time, scientists have recognized the crucial role these chemical signals play in ants' lives, but now Zwiebel and his collaborators are making major advances in deciphering the molecular genetics of ant olfaction. This deeper level of understanding may not only provide new insights into how ants, honeybees and other social insects create and manage complex societies but it may also provide insight into how other "more advanced" animals do so as well. At the same time, it could produce more effective methods for keeping ants out of the kitchen and off the picnic table.

"Ants are unique in the insect world because they have more than 400 odorant receptors compared to 60 to 80 in other insects like fruit flies and mosquitoes," said Jesse Slone, the former Vanderbilt University research associate (now research fellow at the University of Cincinnati Children's Hospital Medical Center) who performed much of the work. "The receptors are arranged in 24 different subfamilies. We selected 25 odor receptors from a number of these groups and decoded them—exposed them to a battery of different chemicals and determined the ones they respond to."

One of the goals of the research was to test a hypothesis that the Vanderbilt biologists made five years ago when they discovered that ants have the highest number of odor receptors identified in any insect species to date. They suggested that the purpose of a highly-expanded subgroup (labeled the 9-exon family) of these extra receptors might be specifically to identify and decode the chemical signals ants use to regulate their complex social behavior. To test this idea, the researchers measured the response of the ant receptors to the specialized class of chemical compounds that coat the ants, called cuticular hydrocarbons. But they also measured their response to other common odorants that scientists know affect other insects.

"It turns out that it's not that simple," said Zwiebel. "The ants appear to be using all the chemosensory tools they have for sensing all the different types of scents that are important to them."

Now that they have an effective method for decoding the ants' odorant receptors (developed by collaborators Gregory Pask and Anandasankar Ray at the University of California, Riverside), the biologists can now begin deciphering the chemical codes that the insects use to communicate.

"These complex blends of hydrocarbons are the crux of ants' social structure. They use them to identify intruders, for foraging, for nursing, for all sorts of behaviors. The next step is to begin associating different chemical signals with specific behaviors," said graduate student Stephen Ferguson, who is attempting to identify the chemical signals that trigger aggression.

Several years ago, while studying mosquito olfaction, the Zwiebel lab discovered a new class of insect repellent that is thousands of times stronger than DEET, the most common active ingredient in commercial insect repellents. The compounds (known as VUAA) work by over-exciting the mosquito's sense of smell by turning on a specialized odor co-receptor channel that must partner with all other odor receptors for them to work.

"It's like being closed in an elevator with someone wearing way too much perfume. If it overwhelms your sense of smell, the net result is repellence. We call this 'excito-repellency,'" said Zwiebel.

It turns out that ants (and indeed all insects) have the very same co-receptor and the Vanderbilt biologists have determined that the excito-repellent compounds work equally well with ants as they do with mosquitoes.

The PNAS article is one in a series of co-authored, peer-reviewed papers on ant olfaction being published in coordination by a network of researchers who have been collaborating as part of an interdisciplinary project titled "Epigenetics of Behavior, Longevity and Social Organization in Ants." The project, which formally ended in 2016, was funded by the Howard Hughes Medical Institute. The other papers describe additional studies on the 9-exon family of receptors as well as the generation and characterization of ant mutants genetically modified to remove their ability to use their odorant receptors. Taken together, these new studies on ant chemical communication have begun to unravel the molecular basis for what the noted Harvard biologist E O. Wilson observed was "the most complicated social organization on earth next to humans."