Secrets charges against a Foreign Office civil servant were dramatically dropped at the Old Bailey yesterday after it emerged that senior figures within his own department had privately admitted no harm was done by his leaking a series of Whitehall documents.

The case against Derek Pasquill, who faced jail for passing secret papers to journalists, collapsed as it was becoming increasingly clear that it could have caused the government severe political embarrassment.

The leaked documents related to the US practice of secretly transporting terror suspects to places where they risked being tortured, and UK government policy towards Muslim groups.

Pasquill, 48, an official in the FCO's Engaging with the Islamic World Group, was arrested two years ago, but only charged last September with leaks to the Observer and New Statesman magazine. He faced six counts under the Official Secrets Act, accused of leaking documents about what Britain knew of America's policy of extraordinary rendition and guidance about which Muslim organisations ministers should embrace.

Yesterday, Mark Ellison, counsel for the government, told Judge Peter Beaumont, the Recorder of London: "There is no longer a realistic prospect of a conviction in this case." He indicated that internal FCO papers revealed that senior officials privately admitted that, far from harming British interests, Pasquill's leaking of the documents had actually helped to provoke a constructive debate.

The prosecution would have had to prove that leaking the documents had caused damage.

The internal FCO papers that fatally undermined the case are understood to have been written shortly after Pasquill's arrest two years ago, but the police and prosecution lawyers had not been aware of their existence until last month and the defence was not aware of them until yesterday morning.

Sources familiar with the case said several ministers were aware that they could be called by the defence. They included the former communities secretary Ruth Kelly, her successor, Hazel Blears, and David Miliband, the foreign secretary.

Julian Knowles, defending, yesterday told the court that the documents should have been released earlier, saving Pasquill the stress and worry of a 20-month police special branch investigation. The leaked documents dealt with topics such as "hearts and minds of Muslims", "engaging with Islamists", conversations between the home secretary and the foreign secretary, "detainees", and Egypt's Muslim Brotherhood, Knowles said.

After he was discharged, Pasquill told the Guardian he had suffered a "very unpleasant ordeal", adding: "I am relieved I have now been completely vindicated in my actions exposing dangerous government policy and changing its priorities."

Of his decision to leak the papers, Pasquill, who remains suspended on full pay, said: "I realised that is a dangerous way to proceed, but this was an issue which was obviously of public interest given the circumstances we are in at the moment in the UK and the world."

Neil O'May, Pasquill's solicitor, said: "The prosecution was a scandal. It was a case where the government was shooting the messenger. It is clear that the Foreign and Commonwealth Office took the view that Derek Pasquill's disclosures were not damaging to international relations. Nevertheless, they let [him] suffer an investigation and arrest, a charge and a full criminal prosecution for two years before they disclosed their views."

Lady Scotland, the attorney general, gave her assent to the prosecution before Pasquill was charged, a spokesman said last night. The Guardian has learned that one of the prosecution's arguments was that the leaking of any official documents damaged the UK's relations with the US.

A Foreign Office spokesman said leaking official documents was "absolutely contrary" to good government. "As Mr Pasquill may be subject to internal disciplinary procedures, any further comment would be inappropriate."

The editor of the New Statesman, John Kampfner, described the prosecution as misguided and malicious, "particularly given that a number of government ministers privately acknowledged from the outset that the information provided to us by Derek Pasquill had been in the public interest and was responsible in large part for changing government policy for the good in terms of extraordinary rendition and policy towards radical Islam".

The Observer described Pasquill as "an honourable civil servant who stood up for the best liberal values of his country".

James Welch, legal director of the civil rights group Liberty, said: "No more innocents should suffer before the government honours its promise to review the Official Secrets Act.

Backstory

The paper trail

Martin Bright began receiving dozens of leaked Foreign Office documents in late 2005, which he described as a journalistic goldmine. A Foreign Office civil servant, concerned that the government was engaging with radical Islamic groups, passed him "dozens of emails, position papers and policy discussions".

Derek Pasquill sent the documents to Bright - a former Observer journalist, now the New Statesman's political editor - because he was concerned that the government, and notably Jack Straw, the then foreign secretary, were concentrating on contacts with the Muslim Council of Britain at the expense of other organisations, a policy the government has since changed.

He was concerned about the FCO's policy of secretly engaging with the Muslim Brotherhood in Egypt, an opposition movement. Ministers admitted that the leaked documents helped them raise issues of concern. One of the documents Pasquill leaked revealed how little British officials knew of the US "extraordinary rendition" programme. The practice, said one official, "is almost certainly illegal". Another asked: "How do we know whether those our armed forces have helped to capture in Iraq or Afghanistan have subsequently been sent to interrogation centres? We have no mechanism for establishing this ..."

One document revealed warnings by Sir Michael Jay, then the FCO's top official, that the Iraq war was fuelling Muslim extremism in Britain. British foreign policy, he warned, was a "recurring theme" in the Muslim community, "especially in the context of the Middle East peace process and Iraq".

Jay referred to the "potential underlying causes of extremism than can affect the "Muslim community, such as discrimination, disadvantage, and exclusion".