However this stops working when I specify listen 8080 and redirects to the default nginx web root (as specified in virtual.conf), if I remove that line it works as expected and redirects to the correct directory.

Does anyone know why this is happening and how I can set a non-standard port for my vhost so varnish can listen on port 80.