Are there really 6.4 million fascists, racists or xenophobes in France? Not at all, says the French political class. Yet this question continues to dominate the final stages of the French presidential election campaign.

The Socialist François Hollande topped the first round vote on 22 April ahead of the rightwing president, Nicolas Sarkozy, creating a dynamic for the left. But Hollande and Sarkozy now face each other in a final run-off on 6 May that is far from clear cut.

The far right dominates headlines after the Front National's Marine Le Pen came third, with the party's highest ever score of 17.9%. Her 6.4 million voters now hold the result in the balance.

Sarkozy has little chance of being re-elected unless he wins over a majority of Le Pen's voters. For the unpopular president, things are extremely difficult but, political experts say, not totally impossible.

Analysts say he needs the support of around 80% of Le Pen voters to win. Polls – which vary wildly – show on average around half Le Pen voters voting Sarkozy, 20% choosing Hollande and 30% abstaining or spoiling their ballot. Sarkozy's camp says all is to play for, despite recent polls showing Hollande winning.

Meanwhile, the political class is soul-searching about what the strong Le Pen vote says about France.

Hollande describes Le Pen's voters as "workers who do not know what tomorrow will bring, pensioners who can't cope any more, farmers who fear for the survival of their farms, the young who ask themselves: where is our future?" The Socialists must reach out to Le Pen's blue-collar supporters, an electorate with whom the party had long been accused of losing touch.

Whoever wins the election must govern a country that is fractured and divided, facing record unemployment, spending above its means and with immediate painful and unpopular choices to make about its finances. There will be no honeymoon period, commentators say, and much less if the Front National gains political ground in opposition, gearing up to try to win its first parliamentary seats in decades.

At a rally in Paris suburb of Raincy on Thursday, Sarkozy continued to veer even further to the right to court Le Pen's voters, stressing the need to curb immigration, tighten borders, protect family values, secure "fortress France" and fight the dangers of what he warned was an encroaching multiculturalism.

He vowed he would never let foreigners resident in France vote in local elections, countering a key promise of the left. He made no apology for courting the far right.

"Must I, like François Hollande, hold my nose in front of 6.5 million French?" he asked. "Just because those 6.5 million French want to overturn the table, should we wipe them from the map? … Do you really think they are 6.5 million fascists?"

Hollande has slammed what he called Sarkozy's race to adopt the ideas and language of the Front National. "There must be limits," he said, warning against making immigration "the only subject of the campaign".

The Socialist has refused to veer from his manifesto, insisting changing Europe's tack on austerity and growth is a way of protecting France. Le Pen, who set out to detoxify her party, has claimed "ideological victory", saying Sarkozy had adopted her programme on a number of points.

The language used to describe voters for the Front National, a party which had long been stigmatised by the mainstream as xenophobic, racist and Islamophobic, is key.

Politicians from left and right say Le Pen's score was the product of anti-Sarkozysm, disgust for the political elite, fear for France's place in the world.

"Not all [Le Pen voters] are racist or anti-foreigners," said Étienne Pinte, an MP from Sarkozy's UMP party critical of the president's lurch to the right.

"There aren't 6 million racists and fascists in France," said one leftwing MP. Sarkozy said "there may be xenophobia among certain leaders of the Front National, such as Jean-Marie Le Pen", but he hadn't heard it from Marine Le Pen.

In an open letter to Hollande and Sarkozy on Thursday, Le Pen defended the "honour" of her voters and said it was unacceptable if her supporters were branded "xenophobes".

• This article was amended on 27 April 2012. The headline was changed from "Far right voters to play key role in final stage of French election campaign" to reflect its more featurey, analytical tone.