According to http://taxes.about.com/od/backtaxes/qt/irs_tax_penalty.htm the interest rate is 4% per year (not bad) but the penalty is 0.5% until the amount is paid in full. It's not clear to me if that is recalculated as you pay it down, or if it's calculated as the starting balance times the total lateness (date of last payment).