WASHINGTON—The Obama administration is teaming up with some conservative lawmakers in Congress to try to restore voting rights to millions of ex-convicts whose criminal records bar them from casting ballots.

Attorney General Eric Holder said the remaining laws in 11 states that restrict felons from voting are outdated, racially unfair and counterproductive. Mr. Holder noted that in Florida, such laws prevent roughly one in 10 people...