President Obama’s Middle East strategy is in ruins and the west is paying the price of having its bluff called, writes John Bew.

Liwa Tahrir al-Sham rebels carry away the body of a comrade from the Jobar front line in the suburbs of Damascus. Photograph: Laurent van der Stockt/Reportage by Getty Images.

President Obama’s thinking about foreign affairs is deep, reflective and nuanced, and not without a moral compass. But it has been severely tested by events in Egypt, and ultimately exhausted by Syria. His attempt to reconcile a broadly liberal world-view with a realist understanding of the limits of American power has been admirable but has left him with an increasingly frayed and incoherent strategy in the Middle East – perhaps no strategy at all.

In 2007 Barack Obama told the New York Times that one of his favourite philosophers was Reinhold Niebuhr, a theologian and subtle commentator on foreign policy who advocated US intervention against the evil of Nazism. He later became known as a supporter of “containment” during the cold war.

“God grant me the serenity to accept the things I cannot change,” Niebuhr wrote in 1943, “the courage to change the things I can and the wisdom to know the difference.”

What Obama claims to have learned from Niebuhr are two core notions that might be taken as bookends to his present approach to the war in Syria – beginning with his strongly held position of non-intervention and culminating in the military response that the US looks likely to pursue in the course of the coming weeks.

On the one hand, Obama argued, Niebuhr recognised “the compelling idea that there’s serious evil in the world, and hardship and pain” but thought that “we should be humble and modest in our belief we can eliminate those things”. Humility in the exercise of power has been the keynote of Obama’s post-Bush approach to foreign affairs; as he reiterated in an interview on CNN, the US cannot solve the conditions that have caused the Syrian civil war.

On the other hand, however, he also stated his conviction that “we shouldn’t use that as an excuse for cynicism and inaction. I take away [from Niebuhr] . . . the sense we have to make these efforts knowing they are hard, and not swinging from naive idealism to bitter realism.”

As every legal and philosophical red line has been transgressed in Syria, it will be interesting to see what this sophisticated doctrine looks like in practice. The answer is that it is likely to be messy. Yet the uncomfortable truth is that, with more than 100,000 people dead and most of the Middle East more destabilised than it was by the Iraq war, nonintervention has also proved to be much messier than its advocates, including President Obama, hoped.

Obama has been right about one aspect of the crisis all along: the conflict there is dizzyingly complex, utterly brutal, grounded in centuries of history and fuelled by sectarian and regional divisions, and it cannot be solved by external intervention. Crucially, in this view, he has had the full backing of General Martin Dempsey, the chairman of the US Joint Chiefs of Staff.

Yet something else was made crushingly obvious by the deaths of hundreds of people in the suburbs of Damascus on 21 August, allegedly as a result of a chemical weapons attack ordered by Bashar al-Assad’s government. It was not simply that Assad is prepared to win the civil war at any cost; that much has been obvious since the start of the conflict. It was that the Syrian regime – and perhaps more importantly its allies in Russia and Iran – seemed to want America to be watching as it did so.

One of the most vexed questions when it comes to Syria has been which outcome to the civil war looks worse from a western perspective. There is no shortage of advisers and experts in Washington, DC who calculate that an Assad victory would not be the worst-case scenario, in a country where jihadist groups are increasingly defining the character of the opposition. Until the 21 August chemical attack, they were winning the argument. The Syrian regime’s forces have been growing stronger since April, and with every month that has passed, the chances of western intervention have receded. Like the US military hierarchy, most Americans have supported the non-interventionist stance, giving the president a solid political basis for his position.

Now, however, Assad has denied Obama even the luxury of averting his gaze. In one move, he has done more to put the US president’s Syria policy under the spotlight than ten visits to rebel-held areas by Senator John McCain could ever do. The message that he seems to have sent is that he is not content with winning quietly, as Obama appeared prepared to let him do. He intends to make his victory also a defeat for America’s standing in the region.

Underlining America’s impotence is part of the prize, a premium on which Assad has been set by his sponsors in Tehran and, to a lesser extent, Moscow. Obama’s concern will be that what is happening in Syria is indicative of a trend emerging across the region and that the dynamics of it are already in play with regard to Egypt and Iran.

The timing of the attack was highly significant – so much so, that it would give credence to the theory that it might have been a rebel “false flag” operation, if all the evidence did not point to the regime. It took place two years after Obama said that Assad “must go” and almost a year to the day that he declared: “We have been very clear to the Assad regime . . . that a red line for us is if we start seeing a whole bunch of chemical weapons moving around or being utilised.” By “we” – as hawkish commentators in Washington are reminding the president – he meant not the United Nations, but the United States.

Consider the following and the Syrian leader’s brazenness takes on a broader significance. The chemical attacks took place a 20-minute drive from the UN inspectors who had arrived in Damascus a few days earlier in order to investigate allegations of previous chemical weapons attacks by the regime. This in itself was the tail end of the diplomatic process, rather than a ratcheting up of pressure from the US and its allies.

Just two days before the attack, a White House intelligence official briefed Foreign Policy magazine to the effect that: “As long as they keep body count at a certain level, we won’t do anything.”

Remember, too, that this is not the first time that Obama’s “red line” has been crossed; the US, the UK and the French governments already believed that chemical weapons had been deployed by the regime in the previous few months. Assad’s willingness to dance back over the line again – in the most grotesquely sensational way he could – can only be taken as a calculated escalation of the diplomatic game.

Rather than take advantage of US quietism, Assad and his allies took a gamble on flouting it, and in a manner that would cause longterm damage to American credibility in the region. Such risk-taking may seem counterproductive and irrational to external actors, but it was based on the fact that Obama’s bluff had been called.

The first time that Assad crossed a red line, the US response was tentative and cosmetic and had no impact on events on the ground. It came in the form of an announcement that logistical support would be offered to the increasingly rudderless Syrian National Council. The muted nature of the response from Washington caused the rebel leadership to give up on the prospect of serious intervention from the west, creating more divisions in the opposition and leaving the door open for Assad to intensify his campaign.

Obama is not a naive liberal internationalist. His thinking on foreign affairs is hard-headed and he has demonstrated – in the huge expansion of drone warfare under his leadership – his willingness to take pre-emptive and lethal action in the name of US national security. He is acutely aware that the American public shares his reluctance to assume once again the role of the world’s policeman.

Cornered: Barack Obama is finding that a non-interventionist policy doesn't work without a credible threat of force. Photograph: Pete Souza/Polaris/Eyevine.

It was only recently that Obama commented that Vladimir Putin behaves like “the bored kid in the back of the classroom”. But what does Assad’s boldest stunt yet, which has been followed by the usual choreographed obfuscation from Russia, make Obama look like? The kid in the front of the classroom who wants to avert his gaze from the bad boys at the back but keeps getting ink flicked in his hair?

The criticism that is increasingly levelled against the president, from both the left and the right of the US foreign policymaking establishment, is that his approach to international affairs is reactive, dependent on counterpunching, and has no strategic vision. His “big-tent” approach to the making of foreign policy – housing an eclectic range of views, from the staunchly realist secretary of defence, Chuck Hagel, to his liberal interventionist ambassador to the UN, Samantha Power – has clogged the decision-making process and prevented the emergence of coherent policies.

Each important decision – to extricate the US from Iraq, the “surge” in Afghanistan, the intervention in Libya, the response to the fall of Hosni Mubarak in Egypt and the procrastination over Syria – has been played out through a series of struggles inside his administration, characterised by leaks, personality clashes and long delays. In all of this, the president has never shirked responsibility for making the final decision, but neither has he “led from the front” or set the agenda with a clear world-view.

Moreover, when it comes to the power struggles engulfing the Middle East, the US has been torn between a set of undesirable outcomes for the past three years. A glimpse of the most desirable scenario – the success of secular, liberal, democratic revolutions – has come and gone. However, vacillation by Washington about what constitutes the least bad endgame, particularly in Syria and Egypt, has opened the door for others to enforce their vision and interests.

As the moral urgency of the Syria crisis intensifies, even the selfish strategic justifications for non-intervention do not look convincing. The least persuasive objection has been Defence Secretary Hagel’s suggestion that military intervention “could hinder humanitarian relief operations”. General Dempsey’s line that “the use of US military force can change the military balance, but it cannot resolve the underlying and historic ethnic, religious and tribal issues that are fuelling this conflict” was more to the point.

Equally, the prospect of handing a victory to some of America’s most ardent sworn enemies – who increasingly dominate the ranks of the Syrian opposition – provokes an understandable neurosis. American involvement, according to Zbigniew Brzezinski, the national security adviser under President Jimmy Carter, “would simply mobilise the most extreme elements . . . against the US and pose the danger that the conflict would spill over into the neighbourhood and set Jordan, Iraq and Lebanon on fire”.

Yet this opinion is in danger of looking like a self-fulfilling prophecy. All of these things are already happening. It is hard to know how Syria could get any worse but it keeps on doing just that.

Increased pressure has come from America’s two chief allies on Syria: France and the UK. “If it is proven, France’s position is that there must be a reaction,” said Laurent Fabius, the French foreign minister, after the attacks. Although a ground invasion is still off the table, Fabius made it clear that the action would entail military “force” of some kind.

William Hague’s change of tone since the chemical attack seems to indicate a willingness to take up the gauntlet thrown down by Assad. “We, he United States, many other countries including France, are clear that we can’t allow the idea in the 21st century that chemical weapons can be used with impunity,” he told the Today programme on 26 August. Diplomacy had failed.

For the first time, the Foreign Secretary and the Prime Minister think they might be able to gain sufficient support for substantive international action on Syria. Both have long wanted to do more; this is the pretext that might also allow them to win a vote in parliament for a limited military option such as precision air strikes.

It was American oversight, scrutiny and leverage that prevented the Egyptian army from using excessive force against the civilian protesters who brought down Hosni Mubarak in 2011. It was partly American dithering about the army’s counter-revolution that led the Egyptian military command to calculate this summer that it could get away with massacring Muslim Brotherhood supporters of the ousted president Mohammed Morsi.

Meanwhile, others are filling the void and playing idiosyncratic geopolitical games in a way that defies western logic. Saudi Arabia offers financial backing to extreme jihadist elements of the Syrian opposition while supporting the crushing by the Egyptian military of the comparatively moderate Muslim Brotherhood (about which the Saudis are deeply neurotic). Within Syria, Iran and each of the Gulf states – whose interests do not align – are engaged in their own version of the “Great Game”, which is likely to have longterm effects on the region.

Worst still, the chaos in the Middle East is creating ideal conditions for terrorism to flourish in Syria and elsewhere. Islamist grievance narratives against the west have been given their greatest boost since the decision to invade Iraq. The attempt to smother the Muslim Brotherhood in Egypt particularly risks forcing more elements of political Islam underground and into violence as the muted US response fuels a perception that the west is complicit in the process.

If there is one thing the west has learned, it is that prolonged and sustained conflicts that attract international jihadis have longlasting consequences. The emergence of new ungoverned spaces has given such groups the space to train, mobilise and act.

The fear of “blowback” is much more acute in European capitals such as London and Paris because of the relative proximity of the conflict and the flow of European citizens to fight on behalf of the rebels in the Middle East. But given that the Americans are engaged in open-ended drone warfare in Afghanistan and Pakistan, as well as Yemen and parts of East Africa, they would be loath to have to extend such a campaign to rebelheld Syria or Sinai.

It is not these broader concerns that have changed the calculus in Washington, however. Rather, it is a single ghastly event that seems to have sucked Obama into the “intervention trap”, against his better instincts.

The likelihood is that any military action will be limited – probably Tomahawk missile strikes supported by cyber attacks, but with no incursions into Syrian airspace. It will be led by the United States, alongside France and Britain, and will probably take place without UN sanction. There is no prospect of putting troops on the ground. Strikes are likely to be directed at chemical and biological weapons installations over a relatively short period.

Significantly, they are not likely to be intended to destroy the Assad regime and open the door for a rebel victory. In other words, while the official position of the US, France and Britain is to support “regime change”, there is little prospect that they will make this the aim of any military campaign as they did in Libya. How the Syrian regime and its allies respond is difficult to predict.

And so, in effect, the implication of such a campaign is that its parameters have been set by the Syrian regime, even if unintentionally, and the Syrians will know what to expect.

The truth remains that Obama’s “red lines” in themselves were conceived in the absence of a strategy for how to respond to the war in Syria. They were unscripted and speculative and reflected a desire to stay out.

Another lesson from the Syrian conflict is that non-intervention does not work in a strategic vacuum. To be successful, the policy needs to be more than a checklist of arguments against intervention. Counterintuitively, as Britain’s most anti-interventionist foreign secretary, Lord Castlereagh, recognised, it requires a credible threat of force.

As Castlereagh told the House of Commons in 1821, he “should deem it most pusiillanimous conduct on our part, if, after interfering on a question of this nature, we limited our interference to the mere delivery of a scroll of paper, and did not follow it up with some more effectual measures. Were we to turn itinerant preachers of morality . . . and to follow up the doctrines which we preached by nothing else but what was contained in our state papers?”

With deep reluctance, Barack Obama has been forced to reach the same conclusion, but his reticence and equivocation over a long period have left him at the mercy of events. It is hard to lead from behind when you don’t even want to look.

John Bew is reader in history and foreign policy in the war studies department of King’s College London. From October, he will take up the Henry A Kissinger Chair in Foreign Policy and International Relations at the Library of Congress in Washington, DC