While it seems right to only show the partner addresses in invoice and shipping addresses, this fix will prevent to use sibling addresses (the address of another contact in the same company), which is a requirement for several users. Therefore, this fix causes more problems than it resolves.

Moreover, these addresses are displayed on the invoice form as following: Company Name / Partner Name / Address Name. Therefore, if you want to change the address while seeing only the partner addresses, you just have to remove the "Address Name", leaving "Company Name / Partner Name".