David Cameron’s ‘spectacular success’ in negotiating down the proposed EU budget increase – from 6% to 2.9% – exists only in the eye of the beholder. Certainly Brussels’ bureaucrats will have less pork to barrel than they’d hoped. But, to even the most Europhile observer, extra funding in this time of austerity smacks of odd priorities and special treatment. So, did we triumph or were we once more outsmarted by the Eurocrats?

The truth is neither. This wasn’t the victory Cameron spun but nor was it the outright defeat many Conservatives are mourning. It was a diplomatic fudge of the kind that Britain has come to specialise in – a ‘score-draw’ rather than an outright win.

Of course, Cameron ought to be used to this scenario by now. His trip to the US earlier this year brought success in terms of face-time with the POTUS but also an embarrassing about-face. He was forced to change his mind on whether or not he would meet with senior US Senators to discuss the tangled web of interests at the heart of the al-Megrahi release. It was right that he met the Senators – like it is right that he forced downward the proposed EU budget increase – but the way in which Britain acts on the world stage can appear slap-dash and reactive and this limits the ability of the Government to achieve its international objectives. Our diplomatic endeavours are structured to fail; real and urgent reform could improve our standing in the world and place more global influence in the hands of our politicians.

The fault is not with Cameron or with William Hague – both of whom are genuinely persuasive communicators – nor is it really the fault of the Coalition: Britain’s lack of international diplomatic projection is an issue that has long plagued governments of every stripe. At the heart of the problem is not politicians who come and go but the institution that stays the same – the Foreign and Commonwealth Office.

The FCO has many talented and experienced operators at its disposal – representing British interests in a myriad of places and cultures. From Dar es Salaam to Denmark you will find the smooth, consistent hand of British diplomacy steering our friends and our enemies in a particular direction. For many this continuity is the great strength of our diplomatic corps, ageless and constant. For me, it is one of the greatest long-term failings of British politics.

The reason? Well it goes right back to first principles; our men and women in foreign capitals work hard for ‘British interests’ but they are largely unaccountable in terms of what those interests might be. Our diplomats belong to a closed, inaccessible institution that wields enormous power and which holds a definite, group-think ideal of what must be done to promote Britain – one that is often starkly at odds with what British people believe.

Thus, even with our most Eurosceptic premier and foreign secretary in over a decade, the FCO’s representatives in Brussels continue to assume that there will be perpetual budget increases for the machinery of the union – and are startled by the vehemence of opposition at home. Even as Prime Minister Blair made the case for Britain supporting America in its Iraqi intervention – with the backing of Parliament and of the majority of British people at the time – FCO officials tried to use dubious legal opinions and the clout of their influence to prevent action.

The FCO has its own views on what ‘British interests’ are. What is more, because unlike other departments they are used to being left to ‘get on with the job’ with limited interference from above, they often get their own way at the expense of those who are actually accountable to the public. That’s why Cameron struggles to get his voice heard when it comes to technical and complex negotiations like that on the EU budget – the diplomats who should be making his case for him have their own agenda.

It doesn’t have to be this way. In the US senior diplomats are selected by the executive to represent the political will of the American people. This forges a synchronicity between the voice of the President and the voices of his ambassadors – especially as around 30% of all senior US diplomats are recruited from outside the Foreign Service entirely, bringing fresh blood, fresh ideas and new thinking into play. This ensures that the POTUS and the Secretary of State can trust that their representatives will genuinely represent them rather than the vested interests of the civil service.

Of course this system has its critics – those who argue that the appointment (rather than promotion) of so many Ambassadors makes diplomacy a game for sponsors and ideologues rather than the business of honed professionals. But these criticisms spring from a deeply problematic assumption; the view that, in important political spheres such as foreign affairs, there is a technocratic ‘best’ way of doing business that somehow trumps the political will of elected leaders.

And it is worth noting that the American system does not simply gift sycophants and cronies with new careers in international relations. Look at the US Ambassador to London, who served the Reagan administration and pursued highly successful careers in banking and the law before arriving in Grosvenor Square on behalf of President Obama. He brings with him a wealth of expertise, experience and bi-partisan public service, all of which make him a hugely valuable advocate for America. That mix of private and public achievement is unlikely to found on a British Ambassador’s CV as the vast majority of them are career civil servants who have spent their lives being shaped by government bureaucracy.

Nor do senior American diplomats get posted purely on the whim of the President. They are vetted by the Senate Foreign Relations Committee – a powerful body of senior elected officials – and must be endorsed by a full Senate vote. This process is not merely an exercise in rubber-stamping – as John Bolton discovered to his disappointment when he failed (twice) to be approved as President Bush’s representative to the UN. The role of the Senate in approving senior diplomats for their postings ensures another layer of democratic control over who becomes the face of America in distant lands.

Britain needs reform to its ambassadorial appointments system to replicate the success of political, outside appointments in the US. Our highest representatives overseas should be absolutely in tune with the goals and the vision of our democratically elected leaders at home: at the moment they are not. The FCO promotes a particular view of ‘British interests’ that is sometimes at odds with the view of the British people and of our leaders; that fixed institutional view often wins out because of a culture of internal promotion that rewards conformity to it. By increasing political oversight – through Ministers and through the Committee system – and bringing more people in from outside, we can make our representatives more representative and project a stronger voice for Britain around the world.