This paper is concerned with the stability of a planar traveling
wave in a cylindrical domain. The equation describes
activator-inhibitor systems in chemistry or biology. The wave has
a thin transition layer and is constructed by singular
perturbation methods. Let $\varepsilon$ be the width of the layer. We show
that, if the cross section of the domain is narrow enough, the
traveling wave is asymptotically stable, while it is unstable if
the cross section is wide enough by studying the linearized
eigenvalue problem. For the latter case, we study the wavelength
associated with an eigenvalue with the largest real part, which is
called the fastest growing wavelength. We prove that this
wavelength is $O(\varepsilon^{1/3})$ as $\varepsilon$ goes to zero
mathematically rigorously. This fact shows that, if unstable
planar waves are perturbed randomly, this fastest growing
wavelength is selectively amplified with as time goes on. For this
analysis, we use a new uniform convergence theorem for some
inverse operator and carry out the Lyapunov-Schmidt reduction.