I was getting the same error message while trying to connect through a firewall... I used port forwarding through ssh to get around it... it may be that you are unable to route UDP packets both ways... you can edit vncserver (it's a perl script) to look at what ports it will be using (and change them as needed)