(Note, this is a preprint of an
article published with some editorial changes in 1998 in Calow, P. (ed.)
Encyclopaedia of Ecology and Environmental Management. Blackwell Press.
pp. 379-380)

isolating mechanisms The reproductive characteristics which prevent
species from fusing. Isolating mechanisms are particularly important in
the biological species concept, in which species of sexual organisms
are defined by reproductive isolation, i.e. a lack of gene mixture.
Two broad kinds of isolating mechanisms between species are typically distinguished,
together with a number of sub-types (modified from Mayr 1970):

1) Pre-mating isolating mechanisms. Factors which cause species
to mate with their own kind (assortative mating).

a) Temporal isolation. Individuals of different species
do not mate because they are active at different times of day or in different
seasons.
b) Ecological isolation. Individuals mate in their preferred
habitat, and therefore do not meet individuals of other species with different
ecological preferences.
c) Behavioral isolation. Potential mates meet, but choose members
of their own species.
d) Mechanical isolation. Copulation is attempted, but transfer
of sperm does not take place.

a) Gametic incompatibility. Sperm transfer takes place,
but egg is not fertilized.
b) Zygotic mortality. Egg is fertilized, but zygote does not
develop.
c) Hybrid inviability. Hybrid embryo forms, but of reduced viability.
d) Hybrid sterility. Hybrid is viable, but resulting adult is
sterile.
e) Hybrid breakdown. First generation (F1) hybrids are viable
and fertile, but further hybrid generations (F2 and backcrosses) may be
inviable or sterile.

An alternative classification of isolating mechanisms contrasts pre-zygotic
isolation (items 1+ 2a above) with post-zygoticisolation
(items 2b-e above). As an example of the application of isolating mechanisms,
the apple-feeding host race of the tephritid fruit fly (Rhagoletis
pomonella) differs from the hawthorn-feeding race in that the apple
race emerges earlier in the year (1a), and each host race preferentially
chooses to rest, lay eggs and mate on its own host plant (1b). On the other
hand, laboratory experiments show that there is little behavioral, mechanical,
or post-mating isolation (1c,d; 2a-e).

The term isolating mechanisms was introduced by T Dobzhansky in the
1930s, and has been popularized in a number of books by E Mayr. Both authors
originally proposed that isolating mechanisms were group traits beneficial
at the level of the species; today, this is generally disbelieved. Recent
authors have pointed out that the word "mechanism" is particularly misleading
as pre-mating and post-mating isolation are likely to evolve
as a by-product of natural selection or genetic drift within species, rather
than as a direct result of their utility as barriers to fertilization and
gene mixing between species (a process known as reinforcement).
A leading critic of the
biological species concept and of the term
isolating mechanisms is HEH Paterson, who argues that species are cohesive
wholes as a result of pre-zygotic sexual signalling within species,
rather than due to isolating mechanisms between species. Paterson therefore
introduced a competing idea of species, the recognition concept
of species, in which isolating mechanisms were replaced by specific
mate recognition systems as an alternative. Unfortunately, the word
"system" has as many group-benefit connotations as "mechanism", and the
recognition concept of species has not gained universal acceptance.

There is also the terminological problem that reproductive isolation
combines traits that reduce gene flow, such as mate choice or fertilization
barriers, with traits that select against genes that have flowed, such
as hybrid incompatibility. Lumping these two antagonistic features is confusing,
since they are unrelated and evolve in very different ways. For instance,
whereas it is conceivable that reinforcement might evolve to reduce
an individual's tendency to mate with another species and produce inviable
offspring, it is almost impossible to imagine that hybrid inviability itself
would evolve as an adaptation. This reproductive isolation terminology
leads also to a muddled use of the term gene flow as the opposite
of reproductive isolation; in other words, gene flow comes
to include not only the flow of genes, but also the effects of any natural
selection on the frequency of such genes within each population.

Perhaps the most fundamental problem with isolating mechanisms (and
specific mate recognition systems) is that species are implied to
be qualitatively different from subspecies, races, or forms by their possession
of these traits. Races cannot, in theory, differ in either type of trait
because only species are defined by their possession. Arguably, by making
species seem qualitatively different from races, these terms have spawned
a number of special models of speciation where geographic isolation,
also known as allopatry, or sudden bursts of evolution in small
founder populations (founder events or punctuated equilibria)
play important roles. Only such unusual conditions were thought to be able
to give rise to new species that differ in isolating mechanisms (or specific
mate recognition systems). In reality, there is little to distinguish
mate choice and disruptive natural selection commonly observed
within species from pre-mating and post-mating isolation
between species; and, indeed, it is hard to distinguish species from races
in many actual organisms (see species concepts).

Most serious research on speciationnow avoids using the term
isolating mechanisms because of these unwanted connotations. Instead, researchers
clearly distinguish between mate choice, hybrid incompatibility
and other forms of reproductive isolation. It is probable that the
term isolating mechanisms is becoming obsolete, and even today is confined
mostly to undergraduate textbooks.