But wait, you get access denied to the farm :O. This is due to a known issue in the double hop of credentials. My server credentials are different to my client credentials and my Powershell is using my client credentials. The solution is
to configure CredSSP authentication.