If you use the Bluetooth connection and your husband doesn’t should not be a problem at all.
If you don’t use Bluetooth the volume control on the phone is pretty handy to adjust quickly for each of you.

It is hard enough being hearing impaired, why add another frustration? Why not get another phone? Right now, if you are not together, one of you doesn’t have a phone. The highest percentage of my calls are to my wife.