This old AP lede explains why Sanford has disqualified himself from public service:

South Carolina Gov. Mark Sanford will cut short a secretive Appalachian Trail hike and return to work Wednesday after revelations he’d been gone for four days with no contact with his staff, wife or state leaders. (Emphasis mine.)

Again, sex isn’t the issue. The issue here is that Sanford was utterly irresponsible — that he disappeared from his job — for days.

Think of it this way. He worked for the people of South Carolina. If an employee of yours just disappeared for four days — and it turned out he was with a mistress in another country (as opposed to being, say, in a hospital) — you might personally forgive him — but would you hire him back?