After the installation, you need to re-login in the graphical session.

I've verified this by removing all packages related to the Ubuntu desktop on an Ubuntu 11.04 Live CD and by installing the server using tasksel. After that, I did exactly what you did: installing ubuntu-desktop by running:

sudo apt-get --no-install-recommends install ubuntu-desktop

You should not start X directly through startx, rather run sudo start gdm.