Then-White House National Economic Council Director Lawrence Summers speaks at the Brookings Institution in March 2009 in Washington. / Mark Wilson, Getty Images

by Tim Mullaney, USA TODAY

by Tim Mullaney, USA TODAY

For Larry Summers, two camps of rivals apparently proved to be one too many.

Everyone knew liberals didn't want the former Treasury secretary to be President Obama's pick to chair the Federal Reserve. They were sore about his remark as Harvard's president that women were less likely to excel at science. And they were suspicious of his role in Clinton-era financial deregulation -- especially the decision barely to regulate financial derivatives at all. That loosed a monster called the credit-default swap that forced the $85 billion bailout of AIG five years ago this weekend, and nearly brought down all of Wall Street with it.

The surprise was the level of skepticism about Summers from his own crowd -- Wall Street types and business economists.

In a USA TODAY survey, 56% of the 42 economists said they preferred Fed Vice Chair Janet Yellen for the top job. Conservatives who favor a good sound business administration wanted the quiet, liberal academic Yellen because she's closely identified with easy-money policies that have served Wall Street well.

Either camp of opponents may not have been able to kill Summers' candidacy. The combination of the two made it untenable, as Summers withdrew his name Sunday.

In his letter to the president, Summers wrote of wanting to avoid a contentious confirmation fight. Such a fight became all the more certain when three Senate Democrats (Jeff Merkley of Oregon, Ohio's Sherrod Brown and Montana's Jon Tester) signaled they would vote against him.

Generally, it has been thought Summers favored less monetary accommodation than Yellen has - which cost him support that might have offset liberal opposition. With Wall Street still nervous about the upcoming process of weaning the economy off reliance on the Fed's easy-money policies, they liked Yellen because they knew Yellen.

Yellen is "extremely well-qualified and she's been there through the (financial) crisis," Mesirow Financial chief economist Diane Swonk said in July. `"We need continuity."

Bottom line: When the financial market players line up behind the Berkeley professor -- and the professor would be the first woman Fed chair to boot -- there was no constituency left for a Summers pick. He always appeared to be the President's idea, preferred by too few others.