Based upon a recent email to the X.Org developers' mailing list, Canonical is nearing the point of one of their goals for Ubuntu 10.10 of a rootless X Server, or being able to run the X.Org Server without root privileges.

X11 performance for me is fine with good drivers. Most people are really just pissed about slow drivers. X itself is just fine, performance-wise. If drivers properly accelerated all the things they should, I bet the complaints about X being slow would go away overnight.