Now that home values are rebounding, it will benefit white households much more. Their home ownership rate is close to 75%. Black home ownership is about 45% and for Hispanics, it's 47%.

"Due to disproportionate loss of home ownership among people of color, the racial wealth gap is likely to grow further as families that have lost their homes will see no benefit from the recovering home prices," said James Carr, senior fellow with the Center for Economic Progress.

Minorities also haven't been able to get back into the housing market because banks have made it tougher and costlier for risky borrowers to get mortgages.

Minority families are facing several barriers to becoming home buyers again.

Most home loans are now backed by Fannie Mae, Freddie Mac and the Federal Housing Administration. All have made loans more expensive for borrowers with low credit scores.

Additional fees for buyers with low credit scores can add up -- more than $100 a month when compared to someone with a high credit score getting a conventional loan.

Even worse, FHA borrowers now have to pay mortgage insurance premiums for the entire term of the loans, even during the last few years when the risk of default is very small.

Government programs favor the well off over the poor, said Ratcliffe.

"The federal government spends billions in tax credits to promote long-term asset growth," she said. "Tax-free retirement savings, the mortgage interest deduction and other programs primarily benefit high income Americans.

The wealth gap could widen if the pathway to home ownership is blocked for too many minorities, said Lewis Ranieri, founder of Ranieri Partners.

"We need a system that funds first time homeowners and trade up buyers and it shouldn't be only for middle class whites," he said.