Allegedly shows militants of the Islamic State of Iraq and the Levant (ISIL) taking position on a street in the northern Iraqi city of Samarra Photograph: -/AFP/Getty Images

Following the bulk of western reporting on the Iraq crisis, you'd think the self-styled 'Islamic State of Iraq and Syria' (Isis) popped out of nowhere, took the west completely by surprise, and is now rampaging across the Middle East like some random weather event.

The reality is far more complex, and less palatable. The meteoric rise of Isis is a predictable consequence of a longstanding US-led geostrategy in the Middle East that has seen tyrants and terrorists as tools to expedite access to regional oil and gas resources.

Since the second world war, as British historian Mark Curtis documented extensively in his seminal study, The Ambiguities of Power, US and UK goals in the Middle East have focused on oil. As a secret British document from 1958 explained:

"The major British and other western interests in the Persian Gulf [are] (a) to ensure free access for Britain and other Western countries to oil produced in States bordering the Gulf; (b) to ensure the continued availability of that oil on favourable terms and for surplus revenues of Kuwait; (c) to bar the spread of Communism and pseudo-Communism in the area and subsequently to defend the area against the brand of Arab nationalism."

While Saddam Hussein was fighting Iran abroad, not to mention gassing Kurds and Shi'ites at home using the vast quantities of chemical and biological weapons sold to him by the US, Britain, France, Germany, among others, he was our man: In 1988, when Saddam's forces were strafing Halabja with mustard gas and nerve toxins, massacring 5,000 civilians, US imports of Iraqi oil had rocketed to 126 million barrels – essentially one out of every four barrels of Iraqi oil exports. This was a special relationship. US oil companies received a discount of $1 per barrel below prices charged to European companies.

That special relationship only changed when Saddam's anti-Americanism got the better of him. At an Arab summit in February 1990, the Ba'athist leader declared: "If the Gulf people and the rest of the Arabs along with them fail to take heed, the Arab Gulf region will be ruled by American will." He complained that the US would dictate the production, distribution and price of oil, "all on the basis of a special outlook which has to do solely with US interests and in which no consideration is given to the interests of others."

So perhaps western officials thought they were being clever when they encouraged Kuwait to conduct what Henry Schuler – then head of the energy security programme at Washington DC's Center for Strategic and International Studies (CSIS) – described as "economic warfare" against Iraq.

Citing the king of Jordan among other high-level sources, the late investigative journalist Michael Emery reported at the time in Village Voice that Kuwait:

"… had enthusiastically participated in a behind-the-scenes economic campaign inspired by western intelligence agencies against Iraqi interests. The Kuwaities even went so far as to dump oil for less than the agreed upon OPEC price... which undercut the oil revenues essential to cash hungry Baghdad. The evidence shows that President George Bush, British prime minister Margaret Thatcher, Egyptian president Hosni Mubarak, and other Arab leaders secretly cooperated on a number of occasions, beginning August 1988, to deny Saddam Hussein the economic help he demanded for the reconstruction of his nation."

In the runup to the 2003 invasion, oil was again centre stage. While the plans to invade, capture and revitalise Iraq's flagging oil industry with a view to open it up to foreign investors were explored meticulously by the Pentagon, US State Department and UK Foreign Office – there was little or no planning for humanitarian or social reconstruction.

Opening up Iraq's huge oil reserves would avert what one British diplomat at the Coalition Provisional Authority characterised as a potential "world shortage" of oil supply, stabilising global prices, and thereby holding off an energy crunch anticipated in 2001 by a study group commissioned by vice president Dick Cheney.

Simultaneously, influential neoconservative US officials saw an opportunity here to pursue hair-brained ambitions to re-engineer the region through the de facto ethno-sectarian partition of Iraq into three autonomous cantons: a vision that could not be achieved without considerable covert violence.

According to US private intelligence firm Stratfor, Cheney and deputy defense secretary Paul Wolfowitz co-authored the scheme, under which the central and largest part of Iraq populated mostly by Sunnis (including Baghdad) would join with Jordan; the Kurdish region of northern and northwestern Iraq, including Mosul and the vast Kirkuk oilfields, would become its own autonomous state; and the Shi'a region in southwestern Iraq, including Basra, would make up the third canton, or would join with Kuwait.

Stratfor warned presciently that: "The new government's attempts to establish control over all of Iraq may well lead to a civil war between Sunni, Shia and Kurdish ethnic groups… The fiercest fighting could be expected for control over the oil facilities" – exactly the scenario unfolding now. Fracturing the country along sectarian lines, however, "may give Washington several strategic advantages":

"After eliminating Iraq as a sovereign state, there would be no fear that one day an anti-American government would come to power in Baghdad, as the capital would be in Amman [Jordan]. Current and potential US geopolitical foes Iran, Saudi Arabia and Syria would be isolated from each other, with big chunks of land between them under control of the pro-US forces.

Equally important, Washington would be able to justify its long-term and heavy military presence in the region as necessary for the defense of a young new state asking for US protection - and to secure the stability of oil markets and supplies. That in turn would help the United States gain direct control of Iraqi oil and replace Saudi oil in case of conflict with Riyadh."

The Stratfor report noted that the plan was only one among several under consideration at the time, and not yet finalised.

The strategic thinking behind arming both sides was alluded to by one US Joint Special Operations University report which said: "US elite forces in Iraq turned to fostering infighting among their Iraqi adversaries on the tactical and operational level." This included disseminating and propagating al-Qaeda jihadi activities by "US psychological warfare (PSYOP) specialists" to fuel "factional fighting" and "to set insurgents battling insurgents."

This short-sighted divide-and-rule strategy went nowhere within Iraq beyond fueling sectarianism, but has played out across the region. As I previously wrote in the Guardian and elsewhere, both the Bush and Obama administration have - through Saudi Arabia, Qatar and other Gulf states - fostered extremist Sunni groups affiliated to al-Qaida across the Middle East to counter Iranian influence.

That has included extensive financing of jihadist groups in Syria to the tune of up to a billion dollars – a policy that began as early as 2009, and continued in the context of pipeline geopolitics. The US and UK had apparently decided that a proposed Iran-Iraq-Syria pipeline would undermine the interests of their favoured friends – Saudi Arabia, Qatar, Turkey and Jordan.

"The geographic area of proven oil reserves coincides with the power base of much of the Salafi-jihadist network…. For the foreseeable future, world oil production growth and total output will be dominated by Persian Gulf resources."

One strategy to protect US access to Gulf oil explored by the report was "Divide and Rule", which would involve "exploiting fault lines between the various Salafi-jihadist groups to turn them against each other and dissipate their energy on internal conflicts." The US could also concentrate "on shoring up the traditional Sunni regimes in Saudi Arabia, Egypt, and Pakistan as a way of containing Iranian power and influence in the Middle East and Persian Gulf."

This might end up empowering Islamist terrorists, the report recognised – but that could be a good thing as it "may actually reduce the al-Qaida threat to US interests in the short term" (never mind the long term) as they would target "Iranian interests throughout the Middle East and Persian Gulf while simultaneously cutting back on anti-American and anti-Western operations."

The potential results were anticipated. In February, director of the US Defense Intelligence Agency (DIA) Lt Gen Michael T Flynn testified in Congress that ISIS "probably will attempt to take territory in Iraq and Syria to exhibit its strength in 2014."

Now Iraqis are paying the price yet again for our ill-conceived imperial hubris, and the US is desperately considering an alliance with arch-enemy Iran to stave off Isis, whose bloody rampage across Iraq threatens to disrupt Iraqi oil production. The conflict has already triggered price spikes that could worsen if Isis expands its hold of key cities.

A new intervention to keep the lid on oil prices is clearly tempting for the US and UK governments, except this would merely strike at the head of the hydra - the symptom - not the root cause. And so far, self-serving wars for oil are precisely what got us here. The rise of Isis – a movement so ruthless even their parent network al-Qaeda disowned them - is blowback from the same brand of oil addicted US-UK covert operations we have run for decades.

If we really wanted to shut down Isis and its ilk for good, we could start by dismantling and disentangling ourselves from the geopolitical and financial infrastructure of oil hegemony that incubates terror. In the current context, bombs promise nothing more than the road to escalation.

In Einstein's words: "The definition of insanity is doing the same thing over and over again and expecting different results."