It appears that you used the livecd kernel/initrd. Getting those to work, while possible would be pretty tricky.

Instead let me suggest that you try grabbing the mini bootloader directly from the repository.

So if you want to install the 64 bit version of natty then get initrd.gz and vmlinuz from this location. Use those files with your the grub config you posted. When you boot up you should get the text mode Ubuntu installer. You can select the install options from the boot menu. I am not sure if the xubuntu-desktop is available. If it isn't, simply install a minimal system and run apt-get install xubuntu-desktop once the base install is done.