"If they are having direct conversations with either political party to ask for a job next year while they're a sitting governor -- do you see that as a conflict?" Garrett asked Yellen.

Yellen paused, then said she would have to speak with her lawyers, and then reiterated that she didn't believe it was a conflict.

Garrett used Brainard's political donation as a way to argue that the Fed is supporting Hillary Clinton by keeping interest rates extremely low so as to not disturb stock markets during the final stretch of the campaign.

"The public increasingly believes that the Fed's independence is nothing more than a myth," Garrett said.

Garrett's prodding comes days after Donald Trump ripped Yellen and the Fed during the first presidential debate.

"We have a Fed that's doing political things...the Fed is being more political than Secretary Clinton," Trump said.

Earlier in September, Trump lambasted Yellen, saying she had created a "false economy" by keeping interest rates so low for such a long time. The Fed put interest rates at zero in December 2008 under former Fed Chair Ben Bernanke to help boost the U.S. economy during the Great Recession.

The Fed just started to lift rates above zero late last year under Yellen. Trump argues that low interest rates are creating a "big, fat, ugly" stock market bubble and that the economy will suffer when the Fed raises rates, which it plans to do in the years ahead.

Yellen adamantly pushed back against Trump at her quarterly press conference last week, saying the Fed has absolutely no political motives and its decisions on interest rates are free of political influence.

"We do not discuss politics at our meetings, and we do not take politics into account in our decisions," Yellen said.

The head of the International Monetary Fund, Christine Lagarde, backed up Yellen on Wednesday.