Let $G$ be a closed, connected Lie group, and let $H$ be a closed (and therefore Lie) subgroup. There is a natural action of $G$ on the space of left cosets $G/H$, for which the stabiliser of $aH$ is the conjugate subgroup ${}^aH:=aHa^{-1}$.

Now let $G$ act diagonally on $G/H\times G/H$. The stabiliser of $(aH,bH)$ is the intersection ${}^aH\cap {}^bH$ of conjugate subgroups. My question is, what can be said about ${}^aH\cap {}^bH$?

I know that ${}^aH={}^bH$ if and only if $ab^{-1}\in N(H)$, the normaliser of $H$ in $G$. Beyond this I couldn't find much via Google. For instance can ${}^aH\cap {}^bH$ be trivial? Or must it be a conjugate of $H$?

Apologies if this is too elementary.

Edit: Thanks for the answers, which show that not much can be said at this level of generality. Now I am looking at a specific example - the icosahedral group $I\cong A_5$ inside $SO(3)$ (where the space of left cosets is the Poincaré sphere). This subgroup has the properties that $I$ is finite, $N(I)=I$ and $I/[I,I]$ is trivial. Does this allow me to conclude that conjugates of $I$ are either equal or intersect in the identity?

More generally, what can be said if I add the assumption $N(H)=H$ to the original question?

Edit 2: I've now asked about the icosahedral group in another question.

If you consider groups whose cardinality is less than or equal to $\aleph_0$ and endow them with the discrete topology, then you get 0-dimensional Lie groups. Now take $G=S_n$ the symmetric group of $\{1,\ldot,n\},$ and $H_i$ the subgroup of the permutations fixing $i,$ for $i=1,\ldots,n$ these are obviously conjugate each other. Trivially $H_1\cap H_2$ is not conjugate to $H_1$. Excuse me if this answer is not what you wanted.
–
Giuseppe TortorellaMar 13 '11 at 10:19

3 Answers
3

Suppose G=SL(2,C) and let H be the stabilizer of a line (so a Borel subgroup). The matrix $$\begin{pmatrix}a&b\\\\c&d\end{pmatrix}$$
in G acts on projective space by
$$ z \mapsto \frac{az+b}{cz+d}$$
The stabilizer of ∞ is the matrices with c=0, a Borel subgroup. The stabilizer of both ∞ and 0 is the matrices with b=c=0, a maximal torus. In particular, a two point stabilizer is abelian (the intersection of two Borel subgroups), and a Borel subgroup is non-abelian. Hence they are not isomorphic.

This is just a connected version of Giuseppe's answer.

If you consider the Borel subgroup to be the Lie group itself, then you get an example where the intersection of the conjugates is trivial. If G is the group of 2×2 matrices with c=0 and d=1 acting on projective space, then the stabilizer of 0 has b=0, and the stabilizer of both 0 and 1 has a=1 and b=0. In particular, G=AGL(1) and H is a maximal torus, and the intersection of two conjugates of H is the identity. When the intersection is the identity, this is called being sharply two-transitive or having a regular stabilizer.

@Mark Grant: More exotic intersections are certainly possible: Take G=Alt(6) wr Sym(2), H=Alt({1..6})×Alt({7..11}), then H is self-normalizing. The conjugates of H are just point stabilizers, and the intersection of conjugates are just 2-point stabilizers. The stabilizer of 11, 12 is Alt({1..6})×Alt({7..10)}≅A6×A4. The stabilizer of 6, 12 is Alt({1..5})×Alt({7..11})≅A5×A5. Similar ideas happen in any wreath product, but this one was chosen so that H is self-normalizing (so that a point stabilizer moves all other points) and perfect. I don't know a connected version of the wreath product.
–
Jack SchmidtMar 18 '11 at 16:43

Let $G=SE(2)$ be the special euclidean group of the plane, and $H_x$ the subgroup stabilizer of $x$, for any $x\in\mathbb{R}^2$.
These latter ones are conjugate each other, but $H_x\cap H_y$ is trivial for any pair of distinct points $x$ and $y$ in the plane.

The motivation for the question is unclear to me, but it seems much too broad to have an interesting answer even if you limit consideration to connected algebraic subgroups of a general linear group (complex or real). As Jack points out, there are easy examples showing various possible outcomes when you intersect a group with a conjugate. The structure theory of semisimple or other Lie/algebraic groups has been extensively studied and makes it easy to find lots of further examples of this type.

For instance, a Borel subgroup intersects a conjugate in at least a maximal torus when the ambient group is reductive; the intersection may be precisely a maximal torus if the Borel subgroups are "opposite". Concretely, this is seen when intersecting the upper triangular and lower triangular matrices in the $n \times$ matrix group: the result is the group of nonsingular diagonal matrices.
At another extreme, intersecting this diagonal group with a typical conjugate by an upper triangular unipotent matrix will typically produce a finite group.

I disagree with your premise. Clearly, if you have a transitive action of a Lie group, it is of interest to study the fixator of two points. This motivates the study of $G/H$ and the intersection of two conjugates of $H$. In the case of a two-transitive action, there is even a very interesting answer available: these actions have been classified by Tits (and Borel). See L. Kramer, Two-transitive Lie groups, for a beautiful exposition and simplified proof.
–
GuntramMar 13 '11 at 14:36

1

@Guntram: There are certainly such well-focused special cases which are interesting, but the question as asked is too broad and the third paragraph there illustrates the need for cautionary examples.
–
Jim HumphreysMar 13 '11 at 17:54