Your question also makes sense for the transfinite continuation of the automorphism tower (see jdh.hamkins.org/everygroup), where one could consider sets of ordinals $I$. Let me add that it is an open question, for finite groups, whether the automorphism tower (up to $\omega$) is eventually periodic.
–
Joel David HamkinsDec 10 '13 at 10:20

@JoelDavidHamkins: It is very interesting. Thank you Prof. Hamkins
–
user43940Dec 10 '13 at 10:38

Note that $G/Z(G)$ embeds in $Aut(G)$. Thus, if $1\in I$ then $G$ is 2-step nilpotent. In this case, one should be able to conclude that $Aut(G)$ is not abelian with a couple of exceptions. Using this, you should be able to prove that, with few exceptions, $I$ should be empty.
–
MishaDec 10 '13 at 11:09

1 Answer
1

I have a partial answer, which would be a complete answer (that is to say a complete classification of all such sets $I$) if we assume Dickson's conjecture in number theory.

If $G$ is non-Abelian, then $Aut(G)$ is also non-Abelian. Specifically, for two non-commuting elements $a,b \in G$, the inner automorphisms $x \rightarrow a x a^{-1}$ and $x \rightarrow b x b^{-1}$ do not commute.

Hence, $I$ can only be either:

The empty set, by letting $G$ be a non-Abelian group;

$\mathbb{N}$, by letting $G$ be the trivial group or some 'ancestor' thereof;

A set of the form $\{ 0, 1, 2, \dots, n \}$ for some $n$.

In the third case, let $P(n)$ denote the proposition that there exists a group $G$ with the property that $Aut^n(G)$ is Abelian but $Aut^{n+1}(G)$ is non-Abelian. We clearly have $P(n) \implies P(n-1)$, by replacing $G$ with $Aut(G)$.

So, we want to show that $P(n)$ is true for arbitrarily large values of $n$.

Lemma 1: If $G$ is a finite non-cyclic Abelian group, then $Aut(G)$ is non-Abelian.

Proof: By the Structure Theorem for finitely-generated modules over a PID, we can express $G$ as a product of cyclic groups of prime power orders. Then, $G$ is non-cyclic if and only if the product contains cyclic groups of orders $p^a$ and $p^b$ for the same $p$. If we can find non-commuting automorphisms of $C_{p^a} \times C_{p^b}$, then we are done, since they would extend to non-commuting automorphisms of $G$.

Without loss of generality assume $a \geq b$. and consider these two automorphisms:

These do not commute. In particular, $f(g(1,0)) = (1,1)$ whereas $g(f(1,0)) = (1+p^{a-b},1)$.

Lemma 2: If $G$ is a cyclic group that does not eventually become the trivial group under repeated iteration of $Aut$, then $G$ will eventually become non-Abelian.

Proof: Induction on the order of $G$. If $Aut(G)$ is non-cyclic, then $Aut(Aut(G))$ is non-Abelian by the previous lemma. If $Aut(G)$ is cyclic and $G$ is not the trivial group, then $Aut(G)$ is smaller than $G$ and we are done by induction.

We know precisely which cyclic groups lead to the trivial group (http://oeis.org/A117729). Any other cyclic group will thus eventually lead to a non-Abelian group.

Assuming Dickson's conjecture, we can find arbitrarily long Cunningham chains of the first kind. These are sequences $(p_1, \dots, p_m)$ of primes where $p_{i+1} = 2p_i + 1$. Note that under the iterated automorphism operator, we get a nice long sequence of cyclic groups:

Also, if $m \geq 6$ then these groups do not belong to A117729, so must eventually lead to a non-Abelian group by Lemma 2.

Consequently, we establish proposition $P(n)$ for some $n \geq m$, and (assuming Dickson's conjecture) this gives us $P(n)$ for all $n \in \mathbb{N}$.

It's not hard to show a partial converse: if we restrict ourselves to finite groups, $P(n)$ is true for all $n$ if and only if there exist arbitrarily long Cunningham chains.

Without Dickson's conjecture, how far can we go? Well, starting with $G$ being the cyclic group of order $361736822347711983585853438$ (twice the largest element of the longest known Cunningham chain), it takes $19$ iterations to reach a non-Abelian group. Hence, $P(n)$ is true for $n \leq 18$.

The second paragraph is wrong. It would mean that G/Z(G) is non-abelian for every non-abelian G which is of course not true in general as witnessed by $Q_8$ for example.
–
Johannes HahnJan 4 '14 at 2:00