This will also appear as similar errors about incompatible security and so forth from various other VNC viewers; the above error is xtightvncviewer on Ubuntu Linux.

The cause of the problem is that Apple's Remote Desktop supports additional features in security that are not part of the normal VNC protocol; to fix it, click "Computer Settings" under Screen Sharing on the Sharing page of System Preferences, and set a password for VNC viewers. Then it will work.