Former Clinton Treasury Secretary Larry Summers says the Federal Reserve has been hesitant to raise interest rates again because the economy just isn’t strong enough.

"The Fed is completely honest [and] totally trying to do the right thing," Summers told CNBC. "Any suggestion that they're behaving politically of the kind we've seen in the campaign, I think, is ludicrous," he said.

Yellen is the "least political person" in Washington, he added. "The political thing to do isn't to be calling for increases in rates."

Summers believes the Fed shouldn’t hike rates anytime soon. "I look at the economy, and unemployment is below target and inflation is below target as well. So that doesn't seem like the time to be stepping on the brakes."

Summers dismissed the notion that the Fed "missed its window" earlier this year to hike rates. "We haven't had a moment when inflation expectations have been anywhere near 2 percent as captured in the market."

Summers has predicted that a Trump presidential victory would be "unsettling" for financial markets.

Summers, who served in two Democratic administrations, told CNBC that the nationalistic forces that led to the vote in Britain to leave the European Union make the "the possibility of a Trump victory more real" because his supporters share similar world views.

"[But] Trump has behaved in so bizarre a way [post-Brexit] it's made those risks less salient than they could have been" for financial markets, Summers said.

The way Trump has conducted himself on the Brexit issue and past controversial remarks he's made about women and minorities makes his candidacy "look deeply disturbing," said Summers, a former economic adviser to President Barack Obama.

While many prominent business figures have been choosing sides in this contentious presidential race, one investment icon bluntly says it doesn't really matter who becomes the next president.

For one, Warren Buffett, chairman and chief executive officer of Berkshire Hathaway Inc, has said a Trump presidency wouldn’t be the blow to U.S. business that some fear, Bloomberg reported.

“If either Donald Trump or Hillary Clinton becomes president, and one of them is very likely to be, I think Berkshire will continue to do fine,” Buffett, 85, said at the company’s annual shareholders meeting in Omaha, Nebraska.

The outcome of November’s presidential election is unlikely to change the fact that the U.S. is a “remarkably attractive place in which to conduct a business,” said Buffett, who endorsed Democrat Clinton at an Omaha rally in December. U.S. companies have enjoyed “terrific” returns on equity despite a sustained period of ultra-low interest rates, he added.

“Twenty years from now, there’ll be far more output per capita in the United States in real terms than there is now. In 50 years, it’ll be far more,” Buffett said. “No presidential candidate or president is going to end that. They can shape it in ways that are good or bad, but they can’t end it.”

Asked how a Trump presidency might affect Berkshire’s business, Buffett replied, “That won’t be the main problem.” He didn’t elaborate.