If you can be mortgage-free, your retirement will also be more secure, Sun said, adding, "You'll have a true asset" for expenses such as long-term care insurance.

Sun recommended you "up your payment just ever so slightly," by 10 percent, 5 percent, 1 percent — however much you can afford.

Sign Up for Our Newsletter Your Wealth

Weekly advice on managing your money

SIGN UP NOW

Get this delivered to your inbox, and more info about about our products and services. By signing up for newsletters, you are agreeing to our Terms of Use and Privacy Policy.

To be sure, people's mortgage bills have become harder to walk away from. The mean value of mortgage debt for people between the ages of 56 and 61 in 2010 was $73,923, compared with just $27,493 in 1992, according to a recent study.

And while you chip away at your housing debt, you need to make sure you're leaving room for other important expenses, Sun said. For example, your retirement savings shouldn't suffer.