Well, you can't blame him for it now; marrying him told him it's OK. You need to talk to him about this. It's not fair to throw away a marriage over something that can be easily changed...unless you're not willing to work on it.