Not that any of the above have to do with the nature of the error, but maybe I would have landed in the results if the post contained those keywords :P

You're probably seeing many 504 errors, because Wordpress is taking too long to generate the content for Nginx to render.

Many results tell you to tweak a bunch of PHP and Nginx timeout params. But the error is that Wordpress is taking too long, it shouldn't take 30 seconds to spit out HTML.

The problem was where my gut was telling me to look. In the plug ins. Just disabling all plug ins and going one by one pointed to the problematic one. Disabling it and looking for an alternative solved the problem.