In “Stress Test,” his account of the financial crisis that hit shelves Monday, former Treasury SecretaryTimothy Geithner says he regrets the Obama administration’s rollout of programs designed to soften the crash of the housing market. But he says there wasn’t a game-changing policy that fully lessened the pain.

Mr. Geithner offered a defense of several crisis-era decisions on housing, but notably steered clear of one: his 2012 decision to sweep away the profits of mortgage-finance giants Fannie Mae and Freddie Mac. The Treasury Department faces a rash of lawsuits by investors who argue that the decision to replace the initial terms amounted to illegal expropriation.

Fannie and Freddie remain one of the last major pieces of unfinished business from the 2008 crisis.

Here’s a look at key passages framing the administration’s struggles with housing and mortgage issues:

Fannie and Freddie didn’t cause the financial crisis:

“Some on the right have tried to blame the crisis on Democratic-led efforts to combat discrimination against low-income and minority borrowers, such as the Community Reinvestment Act of 1977 or Fannie and Freddie’s affordable housing programs. But the erosion in underwriting standards, the rush to provide credit to Americans who couldn’t have gotten it in the past, was led by consumer finance companies and other nonbank lenders that did not have to comply with the Community Reinvestment Act—which, after all, discouraged redlining for nearly three decades before the crisis. These firms took credit risks because they wanted to, not because they had to; they believed rising home prices would protect them from losses, and their investors were eager to finance their risk-taking. Fannie and Freddie lost a lot of market share to these exuberant private lenders, and while they did belatedly join the party, the overall quality of mortgages they bought and guaranteed was significantly stronger than the industry average.”

On the future of the U.S. mortgage market, including Fannie and Freddie:

“We also decided early in the financial reform process that trying to overhaul housing finance in Dodd-Frank would be too heavy a political lift. There was no immediate rush; we knew Fannie and Freddie would be in conservatorship for a while, and that private mortgage lending was too deeply damaged to come back soon….For now, the U.S. government—through Fannie, Freddie and the Federal Housing Administration—remains the dominant force in mortgage finance.

“Eventually, Congress will have to make some tough choices about the mortgage market—not just how to reduce the government’s dominant role, but how to balance the trade-off between safety and accessibility. We should require substantial down payments for borrowers, which would make it harder for some families to become homeowners but would help reduce the risk of the terrible collapses we saw in this crisis. Higher down payment requirements would help serve as shock absorbers for the system—much like capital requirements for financial firms or margin requirements for derivatives investors—limiting the risk of excessive booms by limiting highly leveraged borrowing. And while I believe some kind of government guarantee for mortgage finance is necessary to support lending when the private market retreats during severe recessions, the guarantee should be explicit, more limited in scope, and more expensive, with private actors assuming larger losses to reduce the risks for taxpayers. Powerful real estate and financial interests tend to align with progressive consumer advocates to fight measures that could make mortgage lending more conservative and mortgage credit less affordable, so change will be a challenge, but it can’t wait forever.”

On a silver-bullet for the foreclosure crisis:

“Housing was an impossibly complex issue that didn’t lend itself to simple solutions, and the limitations of our housing programs were a lot easier to identify than they were to fix. We were under intense pressure to improve these programs—not only from our many critics, but from the President, who was deeply unsatisfied with our early results, and constantly pushed us to do better… We were dissatisfied and frustrated, too. Some of our programs were stumbling out of the gate. Others weren’t ambitious enough. We would keep looking for ways to expand their power, reach, and effectiveness throughout the president’s first term.

“If there had been a game-changing housing plan that could have provided much more relief, we would have embraced it. We had some of the nation’s best progressive talent working on housing. We also had powerful incentives to throw everything we had at the problem; the press was killing us and so were our political allies… We tried to do what we could within the constraints we faced. It wasn’t enough. But it was more than most people realized.”

Why a massive loan-principal reduction program wasn’t in the cards:

“We did not believe, though we looked at this question over and over, that a much larger program focused directly on housing could have a material impact on the broader economy. Jan Eberly, the assistant secretary of economic policy, took a fresh look at these alternatives later, and her analysis concluded that even if the federal government had borrowed and spent $700 billion to wipe out every dollar of negative equity in the U.S. housing market—a “principal reduction” program of utopian proportions—it would have increased annual personal consumption by just 0.1 to 0.2 percent. The projected impact on employment was relatively modest, too, amounting to a cost of about $1.5 million of federal spending per job created. By contrast, our auto rescue cost about $14,000 for each of the one million jobs it saved. In other words, even if Congress had authorized the mother of all principal reduction programs, as expensive as TARP and almost as expensive as the Recovery Act, it wouldn’t have changed the trajectory of the recovery.”

On the shortcomings of the Home Affordable Modification Program, or HAMP, the administration’s primary mortgage-relief campaign:

“We ended up requiring a mountain of paperwork for permanent relief, in part to appease critics such as [Neil] Barofsky, [the special inspector general for the $700 billion Troubled Asset Relief Program] who warned that the limited safeguards in our initial proposal were an invitation to fraud; we decided that in this case he had a point. But Larry [Summers, the director of the White House National Economic Council] warned that we were so worried about “false positives,” providing aid to the underserving, that we would allow too many “false negatives,” denying aid to the deserving. He had a point as well….

“By [late 2009], it was clear that HAMP’s reliance on the broken infrastructure of the mortgage servicing industry was a serious problem. This was probably unavoidable; we didn’t have the authority to start up a new government agency or hire thousands of loan specialists ourselves, and even if we’d been able to get the authority from Congress, it would have been a long and messy process. But the servicers, many of them owned by the banks, had little experience modifying loans, and nowhere near the capacity or the resources they would need to modify millions of loans. They had been completely unprepared for the housing crisis, and had laid off staff in droves after the bubble popped. Now we were asking them to conduct a challenging and time-consuming form of triage, and they were terrible at it—slow to hire, slow to figure out how to provide relief, just slow. In fairness, many of the borrowers they were supposed to track down were hard to find and harder to engage; homeowners also struggled to find every required document. But many times incompetent servicers found ways to lose those documents multiple times….”

Explaining how HAMP became the object of intense public scorn:

“I once played a prank on [top adviser] Gene Sperling, pretending I had told a reporter he was the secret architect of HAMP—not just the substance, but the communications and media strategy. My press secretary got so worried that Gene would have a heart attack that she called to spill the beans; that’s how universally reviled HAMP was.”

More direct control of Fannie and Freddie might not have made a huge difference:

“We also tried to push the Federal Housing Finance Agency to pursue a similarly targeted [principal reduction] program for loans backed by Fannie and Freddie. But acting FHFA director Edward DeMarco, a competent but cautious civil servant who did not want to inflame our Republican critics, refused to allow any principal reductions, even though FHFA’s own analysis showed they would save the government money in about half a million cases. It was amazing how little actual authority we had over Fannie and Freddie, considering they were entirely dependent on Treasury’s cash to stay alive.

Some liberals blamed the President for their failure to provide relief, since he could have fired DeMarco at any time. But we couldn’t just appoint a new director on our own….This was frustrating, but I don’t think a more compliant FHFA would have produced a dramatically different result.”

Not a huge fan of rewriting the bankruptcy code to allow write-downs of mortgage debt, or so-called “cram-downs”:

“I didn’t think cram-down was a particularly wise or effective strategy…. The bankruptcy courts were already overwhelmed, and the bill had the potential to push up mortgage costs for all borrowers—though perhaps only slightly given its limited scope—which would have further weakened the recovery.”

About Real Time Economics

Real Time Economics offers exclusive news, analysis and commentary on the U.S. and global economy, central bank policy and economics. Send news items, comments and questions to the editors and reporters below or email realtimeeconomics@wsj.com.