There are three sides to Larry Summers’ failed bid to succeed Ben Bernanke as chairman of the Federal Reserve: politics, policy and personality. Given the hurdles Summers faced in all three areas, it’s a testament to his intelligence and the esteem in which President Obama holds him that Summers got as close as he did to achieving his life-long goal before withdrawing from contention Sunday afternoon.

The politics were always bad for Summers. The architect of Obama’s 2009 crisis management as the President’s first director of the National Economic Council, Summers’ nomination to head the Fed would have been a referendum on the sufficiency of the country’s economic recovery. A Wall Street Journal/NBC News poll released this past week found 52% of Americans disapprove of President Obama’s handling of the economy, not a national mood the President would want to highlight via lengthy on-air debate between Senators and Summers about what he and Obama might have done differently.

On policy, Summers is regarded as brilliant even by those who consider themselves to be his enemies, and having handled crises as both Treasury Secretary for Bill Clinton and NEC head for Obama, he is clearly qualified to run the Fed. Back in 2009, then Treasury Secretary Timothy Geithner, who worked for Summers under Clinton, told TIME that “[Summers’] solutions tend to be government-led, progressive structuring of the market for what he perceives as the greater social good.” It was that approach that clearly attracted Obama, but could have rankled more-conservative Federal Reserve governors.

And the management of other Fed governors is no small issue. Mixed messages from either the Board of Governors or the Federal Open Market Committee spook markets and Main Street investors alike. Which is why Summers’ famously “abrasive” personality (even allies use that adjective) was not a secondary issue in his chances to lead the Fed. Fed governors regularly speak on the record with the media and how they frame and discuss disagreements among their number moves markets like few other interactions. Managing some of the smartest and most powerful economists and bankers in the world isn’t just a matter of good manners at the Fed: it’s crucial to the functioning of the markets.

Ultimately that may have been the deciding factor for Obama on Summers. As much as he may agree with his policy outlook and as much as he may trust and respect his intelligence, there’s no getting past Summers’ famously difficult personality. It’s one thing to have that problem inside the White House, where the President is always the boss. It’s quite another to have it at the Fed where power is much more a matter of consensus.