I want to build an encounter with an interesting hook - the opponent is hard to defeat with anything other than AoOs. That's why I'm looking for ways to somehow deny the PC (it's a solo game) his normal attacks or make the enemy hard to hit/invulnerable. Maybe something like very high AC which is up only on your opponents turn?
An important houserule of mine - enemy is flat-footed when you make an AoO against him.
I assume that #1 suggestion will be a high Dex enemy with high dodge bonuses, because they're not added when flat-footed.
The enemy is supposed to fight melee, but spells for buffing or defense are allowed.
He's going up against a 20th level PC.

Give him an ability that he can activate as an immediate action during his opponent's turn that gives him a very high AC bonus, but only lasts for that opponent's turn. Make sure you give the player a hint on how to defeat him. For instance, when the PC starts his turn, mention that the enemy has his guard up. When it's the enemy's turn, hint that his guard is down. Alternatively, someone (insert stereotypical old man here) reveals to the PC that the enemy is powerful, but has difficulty defending himself while on the attack, and suffers from a bit of tunnel vision in regards to counter-attacks.

Find a way to give him astronomical (as in, something the player won't be able to even get near with their current resources) move speed and make terrain with lots of obstructions, walls, pillars, etc. Also, give him Shot on The Run or Move-By-Attack or something of that nature.
If you prefer an airborne encounter, the same can be done with spells and, for instance, fountains of flame, falling rubble, flying artworks and furniture which are quite at home in high fantasy, towers and walkways... whatever you like.

Now, his every round consists of moving past the players, chopping at one of them (he won't provoke AoO from the one he's attacking, but he will from anyone else he's moving by) and vanish in the nearby obstructions again. Provided he moves fast enough and the terrain is good, the only moment at which the players will have a line of sight or effect to him will be as he shoots past them.
Perhaps supplement with a magical trap or other systematically re-firing effect of Incite so that they can't ready actions, either.

Unfortunately, for your exact needs, you'll need to go at least slightly homebrew, as there are no RAW things that will completely prevent someone from making normal attacks on an opponent.

Personally, my suggestion would be to give him some variety of special boots that give him a permanent, controlled version of blink. Instead of the random chance of him being in the Ethereal plane, instead the boots would put him in the Ethereal plane while standing still, and only solid while moving (say that both feet have to be on the ground for him to be Ethereal, making walking cause him to exit the Ethereal plane). As long as your PCs have nothing that would give them access to the Ethereal plane, he would be invulnerable to them until he moved.

Yeah, in 3.5 you could do something like this by stacking AC boosting stuff that goes away when flatfooted onto the guy (depending on level) so his touch AC is huge (and pretty sure you can't evasion when flatfooted but hey) but in pathfinder I don't think there's enough stuff to do so.

Very tall enemy with tons of armor on his lower body, but none on his upper body. Happens to be standing on some slick substance that you hint - "might make tripping him easier." AoOs as he tries to get up.

Unfortunately, for your exact needs, you'll need to go at least slightly homebrew, as there are no RAW things that will completely prevent someone from making normal attacks on an opponent.

Personally, my suggestion would be to give him some variety of special boots that give him a permanent, controlled version of blink. Instead of the random chance of him being in the Ethereal plane, instead the boots would put him in the Ethereal plane while standing still, and only solid while moving (say that both feet have to be on the ground for him to be Ethereal, making walking cause him to exit the Ethereal plane). As long as your PCs have nothing that would give them access to the Ethereal plane, he would be invulnerable to them until he moved.