My money are on the Sub rogues back in DS. You had a spec which you could not practice unless raiding since you had to rely on procs from Honour Among Thieves. The rotation should, from what I've heard also have been quite unforgiving if you made a single mistake. But GOD the spec did pull some high numbers.

Demo in early Cata probably had the most complex rotation in the game, but it's much easier now. The difficulty of Demo comes with managing fury and the Meta toggle with your cooldowns. The difficulty of Aff comes with managing all your procs and cooldowns to max dot refreshes. Not sure how you can compare the 2. For me, I don't play Aff because I absolutely hate the idea of managing all those random procs. Not sure if that makes it harder than Demo.

I actually think Ret has a really complex priority system with all the different timings. But there's the CLCRET add on which turns it into one of the simpler rotations. I wouldn't enjoy playing Ret very much without CLCRET.

enhan shammy to play correctly as its a lot of thinks to watch. is fire totem/fire ele up?, is unleash weapon ready?, do you have your SS buff? is searing at 5stacks?, is maelstrom at 5 stacks?, is FS on the target?, are your enbues about to fall off?, is your shield about to fall off? and knowing when to use which spell or dump which proc is a lot to track