Sympathy for Jack Straw is not an emotion I would normally feel. But one must, occasionally, change the habits of a lifetime. The release of government correspondence has shown that as minister for justice, Straw originally wanted to exclude the Lockerbie bomber from a prisoner transfer agreement with the Libyans.

Subsequently he changed his mind, stating: "The wider negotiations with the Libyans are reaching a crucial stage, and in view of the overwhelming interests of the United Kingdom I have agreed in this instance the prisoner transfer agreement should be in the standard form and not mention any individual." In defending his change of heart, he pointed out that Libya has been one of just two countries that has voluntarily dismantled its weapons of mass destruction programme.

Straw is not the only minister who has had to balance moral repugnance, or the interests of justice, on one hand, with the national interests of his country and wider questions of international security on the other. Some purists will argue – and doubtless believe – that the interests of justice must never be subordinated to wider political objectives. But governments have to make choices. They do not just deal with theoretical questions or issues of principle. It is a fantasy to believe that any government can, or should, ignore wider national interests when reaching such decisions.

One should not override the interests of justice or ethical considerations lightly, but sometimes the wider national interests would make it perverse to act otherwise. It is fitting to remember, particularly this week, the decision of Roosevelt and Churchill to welcome Stalin as an ally in the battle against Hitler despite Stalin's own monstrous crimes and his collaboration with the Nazis in the dismemberment of Poland in 1939. The supreme national interest had to be winning the war, and in such circumstances my enemy's enemy becomes my friend.

A similar uncomfortable and disturbing conclusion was reached in the Northern Ireland peace process. The decision of both the British government and the Ulster Unionists to work with Gerry Adams and Martin McGuinness, and accept them as coalition partners in the Northern Ireland assembly, stuck in many people's throats and still leaves a bad taste in the mouth.

Few, however, would argue that it was the wrong decision. Northern Ireland is at peace, and a new generation are growing up without the threat of the bullet and the bomb hanging over them.

But while I have some sympathy for Straw's predicament I believe that he and Kenny MacAskill, the Scottish justice minister, got it wrong in releasing Abdelbaset al-Megrahi to Libya. If his release had been a precondition of Libya giving up its support for terrorism and its nuclear weapons programme, their actions would have been justified. But it was not. Muammar Gaddafi had already come in from the cold, and ceased his objectionable behaviour; and he had been rewarded for so doing. Sanctions against Libya had been withdrawn, trade was beginning to flourish, British and Libyan ministers had met, and ambassadors had been exchanged.

Tony Blair's main objective in returning Megrahi was to enhance trade, and help BP in particular. The Libyan British Business Council had warned of the "grave concern" to its members' interests if Megrahi should die in prison.

Straw will argue that these arguments are academic because, in the event, there was no prisoner transfer and Megrahi was released on compassionate grounds. For all practical purposes that is a distinction without a difference.

The British government wanted Megrahi to leave British soil, and did not mind how it was done as long as it was in accordance with legal procedure. They would have supported prisoner transfer, although that would have been a clear breach of their written assurance to the US in 1998 that if the accused was convicted he would "serve sentence in the United Kingdom".

Likewise, MacAskill seems unable to understand that the issue is not whether compassion is sometimes a justification for early release, but whether it is appropriate when the crime is terrorism – when the criminal murdered 270 people and when he has served as few as eight out of 27 years of his sentence.

We will have to live with the consequences of these judgments for some time: not only are the families of the victims heartbroken but the reputation of Scotland and the UK has been seriously damaged, in the US and also worldwide.

Furthermore, the televised welcome for Megrahi on his return to Tripoli and his audience with Gaddafi has heaped further humiliation on Britain as a whole. This has all been suffered merely in the hope that trade with Libya will benefit British companies. The disadvantages so outweigh the potential benefits that the government's strategy appears almost risible.

It has been worse than a crime; it has been a mistake. The masters of realpolitik like Kissinger and Bismarck would never have been responsible for such a sorry mess. Was it a cock-up or a conspiracy? The unfortunate truth is that it has been a cocked-up conspiracy.