"There may have been a thought at the time that it could be part-time," Gillespie said, when Crowley asked him about Romney's leave of absence. "It was not part-time. The Olympics was in a shambles. The International Olympic Committee was going to pull the Olympics from the United States of America, which would have been a huge embarrassment.

"He took a leave of absence and in fact, ended up not going back at all and retired retroactively to February 1999 as a result."

Gillespie said that President Barack Obama and his campaign were trying to shift the narrative away from the slumping economy. He said it was also "sad to see" members of Obama's campaign staff imply that Romney could have committed a felony if he lied on SEC filings.

"We now know that this president will say or do anything to keep the highest office in the land, even if it means demeaning the highest office in the land," Gillespie said.