Ransom money was paid to the Islamic extremists holding British and Italian hostages in Nigeria before British special forces tried to rescue them from their compound, a news agency has claimed.

Part of a €1.2m ransom was paid to release hostages Chris McManus and Franco Lamolinara, who were killed during the raid on Thursday which sparked a diplomatic row.

The ransom talks, in which both British and Italian officials had participated, began with a request for €5m and the release of prisoners, the Mauritanian agency reported, quoting sources close to the extremists. During the talks, questions were sent to the kidnappers for the hostages to answer about their families to prove they were still alive.

The British took a tougher line in negotiations than the Italians and demands eventually settled on €1.2m and no prisoner release, it is alleged. After a downpayment, British and Nigerian operatives were able to follow the extremists back to their hideout, setting up the raid.

The Italian foreign ministry declined to comment on the report. The Italians protested that London failed to inform them of the raid until it was under way President Giorgio Napolitano calledBritain's unilateral action "inexplicable".

Operatives from Britain's elite Special Boat Service and Nigerian soldiers surrounded the kidnappers' hideout on Wednesday, a day before the firefight in which McManus and Lamolinara were killed, the Italian paper Corriere della Sera reported on Saturday. Quoting a Nigerian journalist, Ahmad Salkida, the paper said that, once surrounded, the kidnappers asked to be able to flee the hideout but their request was turned down by soldiers, who demanded they surrender. The kidnappers refused and the raid got under way.

The wife of one of the guards holding the hostages said on Saturday the two men were taken into a lavatory and shot dead during the rescue attempt.

The woman, who gave her name only as Hauwa and said she was 31, cried into her hands as she spoke to Reuters. Hauwa said bullets were fired into the room where she and her husband were staying, killing him.

"After that, there were about six men who came out of the house with the two hostages," she said. "They came into our wing of the compound, pushed the captives into the toilet and just shot them. I screamed."

Nigerian authorities have detained five Islamist militants suspected of involvement in the kidnapping. Two of the men were arrested before the rescue attempt and three at the compound where the raid took place.

Italian secret service officials were first alerted to the raid by British counterparts at 10.15am on Thursday, Corriere della Sera reported. They, in turn, informed Italian PM Mario Monti 15 minutes later. By 12.30, the UK ambassador to Italy, Christopher Prentice, was holding talks with Italian government officials to update them on the operation.

British government officials have said Italy had been told of the possibility of a raid and Corriere della Sera said Italian defence minister Giampaolo Di Paola had been informed the week before that special forces had been deployed to the area.

But politicians across the political spectrum in Italy have demanded to know why Rome was not warned that the raid was about to take place, calling it a "slap in the face".

Italy's predilection for negotiating with kidnappers instead of rescuing hostages through force was shown in Afghanistan in 2007, when it released Taliban prisoners in return for the freeing of an Italian journalist.

One Italian analyst said the country's declining prestige on the world stage during Berlusconi's years in office meant the UK was less inclined to share its plans with Rome.

"Italy is weak internationally and not considered an equal ally," said Nicola Pedde, the director of Rome thinktank the Institute for Global Studies. "We are coming out of a period which demolished Italy's reputation, the effects of which we have already seen during the Libya campaign."

But Pedde said it was also possible that Italy had been consulted about the raid, but had not responded. "I have heard one rumour that the Italian secret services had been told but had yet to inform Monti, and another that the services had passed on the information but had not been told yet how to respond."

Politicians ousted from power when Berlusconi's government crumbled last year have been quick to accuse Monti of letting down Italy on the diplomatic stage.

"With Berlusconi, the hostages came home alive," said Fabrizio Cicchitto, the leader of Berlusconi's People of Freedom party in the lower house of the Italian parliament.

"The English have been taking the piss," said former interior minister Roberto Maroni.