Let $f$ be a homeomorphism of a topological space onto itself.
We recall that a minimal subset for $f$ is a closed invariant subset $F$ such that there is no closed invariant subsets of $F$ under $f$. Equivalently, $F$ is a closed subset for which every orbit is dense. Using Zorn's Lemma it can be shown that such sets always exist.

It is easy to see that $R$ cannot be a minimal subset, or that irrationnal rotations on the circle are.

Is it possible to classify such sets (and/or the associated homeomorphisms), say, in dimension less than two ?
More precisely, given a homeomorphism of the plane, is it possible to classify its possible minimal subsets ?

If it turns out that this is too ambitious, are there results of this type for a suitably well-behaved (but still large) class of dynamical systems ?

2 Answers
2

It is certainly the case that classifying the minimal subsystems of homeomorphisms of compact 2-manifolds presents profound and fundamental difficulties. This is because some very simple transformations, such as analytic diffeomorphisms of the 2-torus, have extremely rich families of minimal sets.

Let $T \colon X \to X$ be a linear Anosov diffeomorphism of the 2-torus. The topological entropy of $T$ is finite and positive but may be arbitrarily large. If a natural number $k$ is specified, then we may find a linear Anosov diffeomorphism $T$ of the 2-torus $X$ such that the shift transformation on $k$ symbols may be homeomorphically embedded into the dynamical system $(X,T)$ as a compact invariant subset. In particular, every minimal subsystem of the $k$-shift embeds into $(X,T)$ as a minimal subsystem.

This is problematic because the $k$-shift has an enormous number of minimal subsystems, all of which will be inherited by the Anosov system. Indeed, the combinatorial version of the Jewett-Krieger theorem implies that every ergodic measure-preserving transformation of an abstract probability space which has entropy strictly less than $\log k$ may be embedded into the $k$-shift as a uniquely ergodic minimal subsystem. In particular, for a linear Anosov diffeomorphism of the 2-torus with topological entropy large enough, every ergodic measurable dynamical system with measure-theoretic entropy up to some threshold arises as a uniquely ergodic minimal subsystem.

This already presents us with an enormous number of minimal subsystems, because for each $h \geq 0$ there exist uncountably many ergodic measurable dynamical systems of entropy $h$ which are not pairwise equivalent. This is then compounded by the fact that some minimal systems of $(X,T)$ will not arise from such an embedding, the fact that the embedding of the abstract ergodic system into the $k$-shift is in general not unique, the fact that the embedding of the $k$-shift into $(X,T)$ is in general not unique, and the fact that the $k$-shift itself has additional minimal subsystems. Indeed, there is a further theorem due to Denker, Grillenberger and Sigmund which implies that for any finite collection of abstract ergodic transformations all having entropy strictly below $\log k$, we can find a minimal subsystem of the $k$-shift which has an embedded copy of each of these transformations as its only ergodic measures.

On the basis of the above considerations I think that a satisfactory classification of the minimal subsystems of homeomorphisms of the 2-torus is improbable!

I don't really answer your question, but the following is too long for a comment.

How do you know that minimal subsystems of your dynamical system always exist? Usually one looks at actions on compact spaces, and then you can use the fact that intersections of descending chains of nonempty compact sets are nonempty. Now the existence of minimal subsystems follows using a straight forward application of Zorn's lemma.
How do you argue in the general case?

If you consider just a single homeomorphism, then in fact you are looking at an action of $\mathbb Z$ on the space. If we restrict our attention to compact topological spaces, then there is something called a universal minimal system for $\mathbb Z$ actions on compact spaces. Every minimal system is a quotient of the universal minimal system.
However, the underlying space of the universal minimal system for $\mathbb Z$ is a subspace of the Stone-Cech compactification of the integers and hence not metrizable.
I don't know whether there is something like a universal metrizable minimal systems.

Also, I have no idea whether you can classify minimal systems if you severely restrict your class of spaces to be subspaces of $\mathbb R$ or $\mathbb R^2$. But you are looking at homeomaorphisms of the whole real line or the whole plane, right? In this case I would guess you can actually say something reasonable about the minimal subsystems.