letsencrypt fails with “Unable to setup, Let’s Encrypt
Please make sure that your site is pointed to
same server on which you are running Let’s Encrypt Client
to allow it to verify the site automatically.”

The site loads on http but I get connection refused on https. A remote port scan shows port 80 open and port 443 closed. I’ve tried with ufw disabled and enabled:

Your OP said “443 ALLOW ANYWHERE” and I’m glad you figured it out, but I’d like to know, for future reference, where port 443 was still closed? You also said you used vestacp to fix it. What exactly did you do, in vestacp to fix it?

For vestacp, I just started with a clean OS install, rand the vesta install and used a free online cert generator (basically a wrapper for letsencript) to generate the cert. https/443 was working out of the box.