Mark Zandi, chief economist at Moody's Analytics, suggested that within the next year private lenders could be much more enthusiastic about reengaging the housing market, which could have a full head of steam by then.

But right now, there is still plenty of concern about the state of the housing market and the need for growth to remain consistent and allow the government to gradually pull out of the mortgage finance business.

Fannie and Freddie were taken over by the government during the 2008 financial crisis.

During the past several years, lawmakers have batted around the idea of winding down the government's involvement in the mortgage finance industry but have yet to embark on a legislative solution.

While an overhaul remains on hold, Mayer said he expects Congress to kick the can down but said, in the meantime, that the Federal Housing Finance Administration (FHFA) could set up a framework to get help the government pull back their support while getting private lenders back into the lending game.

"Inaction in some cases might be a lot better than action," said Susan Wachter, a real estate and finance professor at the University of Pennsylvania's Wharton School.

"It's not an accident the rules aren't there for the private sector, because we're not yet there," she said.

Mayer suggested that Congress, at least, take up a bill sponsored by Senate Democrats Barbara Boxer (Calif.) and Robert Menendez (N.J.) that could help millions refinance.

Housing Secretary Shaun Donovan, who headlined the Tuesday summit, urged support for that legislation and several others he said would bolster the market.

The bill is set for consideration in the Senate after the elections.

But Holtz-Eakin suggested that Congress stop "bombing" the housing market with legislation that hasn't worked.

The panel of experts agreed the politics surrounding the issue of winding down the GSEs has further hampered any chance of making headway.

"Politics have driven policies about the GSEs and that's not what makes sense for a functioning market," Mayer said.

"We need to significantly reduce the government footprint and reduce role of regulators and we will eventually need congressional action to push these things."