Turns out this is entirely my fault as I had not assigned the test user any roles inside CRM (verified by browsing to the site). After assigning some roles the outlook client could then connect and everything seemed happy.

To assign roles to a user open up your CRM website then under “Settings, Administration, Users” open your user and then choose “Roles” down the left had side. Add your new roles in here.