How do you know when your horse reached his full potential?

I think like James that it's a matter of feeling but of course there are so many elements that can influence (the living conditions, the training method, the rider, the trainer...).

I think there are "physical" limits, for example (it's a little bit of a caricature) a draft horse certainly doesn't have a 150cm course. There are also mental limits, some horses will naturally not have the mind to jump a course with a certain height or keep the head cold and listening to his rider during a whole high level dressage test.

The concept of limit is quite difficult to define in horses, for example a horse may seem limited to a certain height with a rider but go much higher with ease with another rider. It's the same for a rider, he may not be able to reach a level with a horse but get there with another.
So I think we can't talk about the limits of a couple.

In my opinion it's first a matter of feeling, a rider must feel when his horse looks in difficulty and he can't overcome them. It is also the role of the trainer to see when there is no more progression possible, and still maybe the progression is no longer possible with the current conditions but maybe with some changes it would be possible!