This has happened to me : keep your correspondence within Amazon messaging system . Make sure to insure your product and ask for signature . And finally email seller-performance@amazon.com and give them a brief description of what happened and what you have done once they know everything is set. However it is true that is against the policy and they have to change the address .

It won’t affect your metrics as long as you still ship on time, etc.
However, it’s actually against policy. The only way to ship to the new address and both be protected and complaint, is to ask the buyer to cancel and re-order with the correct address.