The US attorney general, Eric Holder, has warned that gains of the civil rights struggle hang in the balance in the face of a determined effort by many states to roll back laws ensuring the right to vote.

Holder told a Washington conference on Wednesday that there is a "growing need to protect the voting rights of every eligible citizen" amid a flurry of legislation and executive orders in US states ostensibly to prevent election fraud with measures such as requiring proof of identity in order to vote.

The meeting was told the laws are intended to prevent African Americans in particular from voting because because nearly one in four black people lack photo identification.

A Congressman from North Carolina, GK Butterfield, said that if new requirements for voter identification and proof of citizenship in many states, and restrictions on groups mobilising people to vote, had been in place four years ago they would have been likely to cost Barack Obama the presidency.

Holder said that in the past two years the justice department has challenged "two dozen state laws and executive orders from more than a dozen states that could make it significantly harder for many eligible voters to cast ballots in 2012".

"Despite our nation's long history of extending voting rights to non-property owners and to women, to people of colour, to Native Americans, and to younger Americans, today a growing number of our fellow citizens are worried about the same disparities, divisions and problems that nearly five decades ago so many fought to address," Holder told a meeting of the Congressional Black Caucus and black church leaders.

"In my travels across this country I've heard a consistent drumbeat of concern from citizens who for the first time in their lives now have reason to believe that we are failing to live up to one of our nation's most noble ideals and some of the achievements that defined the civil rights movement now hang, again, in the balance."

Nicole Austin-Hillery, legal counsel for the Brennan Center for Justice, said that up to 5 million voters could be affected by laws already in place or on hold in the face of legal challenges.

"The efforts underway to change [voting] laws are fast and furious," she said. "This is not just about voter ID, there are a panoply of changes that have taken place."

The supreme court has upheld voter ID legislatation and seven states have introduced strict requirements. But 16 southern states require justice department approval for changes to election laws under the 1965 Voting Rights Act because of a history of discrimination. The justice department has used that power to block voter ID laws in Texas and South Carolina as likely to discriminate against racial minorities as well as other people.

Holder also pointed up the example of redistricting in Texas for elections to the state legislature and US Congress. He said that electoral maps "were manipulated to give the appearance of minority while minimising minority electoral strength".

"In December, we objected to South Carolina's voter ID law after finding, based on the state's own data, that the proposed change would place an unfair burden on non-white voters. In March we objected to a photo ID requirement in Texas because it would have a disproportionate impact on Hispanic voters," he said.

Twenty-three other states now require some form of proof of identity in order to be able to vote.

Austin-Hillery said other measures are also likely to have an impact on voting including proof of citizenship in some states and the curbing of early or Sunday voting which many poorer people rely on because they cannot afford to take hours off work on election day.

She said these measures have a disproportionate impact on minorities, old people and students. One in four African Americans and one in five people over 65 lack photo identification. "A lot of Americans say: who doesn't have ID? But 11% of us don't have it," she said.

Republicans in some states claim that the new measures are necessary because voter fraud is so widespread that it is costing the party elections. But a federal investigation and academic studies have not backed up the claim.

Critics, such as Butterfield, say that the real intent is to reduce the black and Hispanic vote which mostly goes to the Democratic party. He compared the new voting requirements to laws passed at the turn of the 20th century that effectively stripped African Americans of the right to vote in southern states and saw black representatives from the south driven from Congress.

"All of this progress that we have made is under assault. There is a right wing conspiracy that is alive and well in this country that is trying to take us back to 1900 and even before," he said. "What they want to do is not take away the right to vote but if black voter participation can be diminished, even by 10%, it will make that critical difference all across the country.

"President Obama won my state in the last election by 14,000 votes. Had we had a voter ID law in North Carolina he would not have won the state of North Carolina and probably could not have won the presidency."

Holder said that some of these laws fall foul of two key parts of the Voting Rights Act – section two, which forbids racially discriminatory practices that deny or dilute the vote, and section five, which requires the justice department to approve changes to voting regulations in the 16 states with a history of institutionalised racial discrimination.

Section five was reauthorised by Congress in 2006 by an overwhelming vote. Since then it has come under concerted attack. Between the passing of the Voting Rights Act in 1965 and 2010 there were just eight legal challenges to section five. In the past two years there have been nine, of which four are still being fought out in the courts.

Holder sald: "Each of these challenges to section five claims that we attained a new era of electoral equality. That America in 2012 has moved beyond the challenges of 1965 and that section five is no longer necessary. I wish this were the case but the reality is that in jurisdictions across the country both overt and subtle forms of discrimination remain all too common and have not yet been relegated to the pages of history."

Sign up for the Guardian Today

Our editors' picks for the day's top news and commentary delivered to your inbox each morning.