Real estate in your IRA? Be careful

FORTUNE — With the housing market showing signs of stabilizing, investors are turning back to real estate. And an increasing number of affluent savers have been using their retirement accounts to purchase homes, rental apartments, and other properties. Done properly, such a strategy can generate good income with modest volatility. But before you dive in, take heed: It’s a time-consuming proposition, and you’ll need to proceed carefully not just to avoid investment losses but also to stay clear of some nasty tax pitfalls.

To buy real estate within a retirement account, you first need to set up a “self-directed” IRA with a custodian. Once you’ve established the IRA, you can then use it to purchase practically any type of real estate, including vacant land, single- and multi-family homes, commercial properties, co-ops, and condos. But you’ll also have to adhere to a slew of IRS regulations.

For starters, “you can’t use real estate in a self-directed IRA for your personal benefit,” says Richard Price, a CPA with Houston-based Cornelius Stegent & Price. If you do, the IRS could disqualify the IRA, in which case you’d owe ordinary income taxes on the entire value of the property, plus a 10% penalty. For instance, you could buy a Miami condo and rent it to third parties, but you couldn’t stay even one night there yourself. Nor could you rent it out or grant free use of it to most close relatives. You couldn’t renovate or actively manage the property yourself. You’re also prohibited from buying real estate for the IRA that you or selected family members already own.

To prevent the IRA from being disqualified, all income generated by the real estate must be paid directly into the IRA and remain there until you retire. If you sell the property, all proceeds must likewise stay in the IRA. Expenses, such as property taxes, insurance, and improvements, must come straight from the IRA; you can’t use personal funds to pay for them, and you can’t deduct them for tax purposes. You’ll therefore need to leave enough cash inside the IRA to keep the property running (for 2013, you can contribute up to $5,500 to your self-directed IRA, or $6,500 if you’re age 50 or older).

If you sell real estate within a regular self-directed IRA, any appreciation on the property will ultimately get taxed as ordinary income when you take withdrawals, at rates currently up to 35%. But if you set up a self-directed Roth IRA (in which your contributions are made with after-tax dollars), all your property gains and withdrawals will be tax-free. That’s assuming, of course, that you have gains. There’s no guarantee that the real estate market won’t tank again. And rising property taxes, unplanned maintenance costs, and deadbeat tenants could also ding your returns. The upside of real estate is big — but so are the headaches.

–A former compensation consultant, Janice Revell has been writing about personal finance since 2000.