Serbia's hopes of fast-track integration into Europe suffered a severe setback when President Boris Tadic was voted out of office in a victory for a more nationalist rival in the latest European election to kick out an incumbent.

With most of the vote counted, Tomislav Nikolic, a former strident nationalist, took the presidency from the pro-western Tadic by two to three percentage points.

The outcome confounded the pre-election opinion polls and expectations in Brussels, and could also condemn Serbia to a period of political paralysis, with a rightwing president "cohabiting" with a centre-left-dominated government. Parliamentary elections a fortnight ago delivered a mixed verdict, with no clear majorities.

Tadic, the west's preferred candidate, failed to win a third presidential term, but could still become prime minister if his Democratic party cobbles together a parliamentary majority. This would portend endless feuding between the head of state and the head of government.

But Tadic promptly conceded on Sunday night, apparently deciding that the election defeat meant he could not be prime minister. Under Tadic, Belgrade recently won a green light to start the long route to EU membership. It had hoped to begin negotiations later this year.

An early start to talks, however, is now less likely, with Nikolic, a former leading light in an extreme nationalist party headed by a war crimes suspect, less likely to make the concessions on the breakaway country of Kosovo that Brussels will deem necessary for opening negotiations.

"Serbia will keep the EU path but also protect Kosovo. Serbia is a modern country – I will co-operate with everyone," said Nikolic.

Known as "the undertaker" due to an earlier career running a funerals company in central Serbia, the new president-elect, 60, was previously a leading light in the Serbian Radical party, the creation of Vojislav Seselj, a warlord from the 1990s currently on trial at the war crimes tribunal in The Hague.

In the war years, Nikolic was a keen advocate of "greater Serbia" – the conquest of and incorporation into Serbia of large tracts of Croatia and Bosnia. But he broke away to head his Progressive party, tempered his views, and shifted to the centre-right, committing to membership of the EU. His party and his constituency are nonetheless more critical of the EU and western integration and more open to overtures from Moscow.

In an extraordinary gaffe which was nonetheless prescient, the heads of the European commission and European council, both in Chicago for the Nato summit, issued a statement congratulating Nikolic on his victory three hours before the polls closed.

They retracted the statement, saying: "The EU statement on the results of the Serbian presidential elections will be published later tonight. Please disregard the email notification you received earlier on this topic."