McAllister, who announced May 3 he would not be a candidate after a surveillance video was leaked showing him kissing a married staffer, said he's been told that perhaps he rushed the decision.

"I can tell you my intent is not to run for re-election," McAllister said during an appearance in Alexandria," according to the Town Talk of Alexandria. "To say, I'm 100 percent sure, I would never box myself in like that."

McAllister, who said he rejected a suggestion from House Majority Leader Eric Cantor, R-Va., that he step down and not complete his first term in office, said he feels buoyed by the support of his wife Kelly and five children and voters in the 5th Congressional District. He said it is a "blessing to live in the 5th District."

McAllister, a businessman and political unknown, pulled off a surprisingly lopsided victory in the November runoff in November to replace Rodney Alexander, R-Quitman. He ran on a platform of bringing family and Christian values to Washington.