Genetic drift can cause big losses of genetic variation for small populations.

Population
bottlenecks occur when a population's size is reduced for at least
one generation. Because genetic drift acts more quickly to reduce genetic variation in small
populations, undergoing
a bottleneck can
reduce a population's genetic variation by a lot, even if the bottleneck
doesn't last for very many generations. This is illustrated by the bags
of marbles shown below, where, in generation 2, an unusually small draw creates a bottleneck.

Reduced genetic variation means that the population may
not be able to adapt to new selection pressures, such as climatic change
or a shift in available resources, because the genetic variation that selection
would act on may have already drifted out of the population.

An example of a bottleneck
Northern elephant seals have reduced genetic variation probably because
of a population bottleneck humans inflicted on them in the 1890s. Hunting
reduced their population size to as few as 20 individuals at the end of the 19th
century. Their population has since rebounded to over 30,000  but
their genes still
carry the marks of this bottleneck: they have much less genetic variation
than a population of southern elephant seals that was not so intensely
hunted.

Founder effects
A founder effect occurs when a new colony is started by a few
members of the original population. This small population size means that the
colony may have:

reduced genetic variation from the original population.

a non-random sample of the genes in the original population.

For example, the Afrikaner population of Dutch settlers in South Africa
is descended mainly from a few colonists. Today, the Afrikaner population
has an unusually high frequency of the gene that causes Huntington's
disease, because those original Dutch colonists just happened to carry
that gene with unusually high frequency. This effect is easy to recognize
in genetic diseases, but of course, the frequencies of all sorts of genes
are affected by founder events.