Zakari Matazu had just got home when a deafening sound filled his ears and part of his wall cracked and fell to the floor, whipping up a cloud of dust. Then came another boom and his legs started shaking.

"I walked outside my house, then I saw people running helter-skelter and people screaming, and at that point my legs could no longer carry me, so I just sat down on the ground," he recalled. "That is when I saw my neighbour Mama Baby, who was screaming and pointing to a building that had been brought to the ground by the bomb, and she was saying that her children were in the rubble."

Matazu, 29, survived the double bomb blast earlier this month in Maiduguri, north-east Nigeria, that killed about 45 people and destroyed seven buildings. It was the latest blow by the terrorist group Boko Haram to shake the foundations of Africa's most populous state.

Boko Haram is believed to be responsible for killing at least 1,300 people in the past two months and more than 130 people in the past week. The radical sect claims ties to al-Qaida and has ambitions to impose sharia law on Nigeria's 170 million people. In Boko Haram's heartland, even the national military is outgunned in what is fast becoming a lesson to the world in how not to tackle an Islamist insurgency.

"What is clear is that they are as ruthless as any Islamist group or terrorists anywhere in the world," said Antony Goldman, a west Africa risk analyst at London-based PM Consulting. "They're quite happy to hit soft targets, including schools. Some in the Nigerian administration expect this to be a problem for another 10 years."

In some ways, the paradox of Nigeria in 2014 captures that of Africa itself. The continent has enjoyed a decade of economic growth and the phrase "Africa rising" has become widespread among investors and journalists. Yet at the same time the past six months have seen conflicts erupt in the Central African Republic and South Sudan, while economic growth has gone hand in hand with deepening inequality.

So it is with Nigeria which, with oil wealth and a decade of annual growth around 7%, is set to overtake South Africa as Africa's biggest economy, with a value close to $400bn. It has been anointed one of the "Mint" emerging economies – along with Mexico, Indonesia and Turkey – by economist Jim O'Neill. Nigerians drink more champagne than Russians do.

But just as Africa cannot shake old habits of instability, so the booming commercial capital, Lagos, is only part of the story in one of the continent's most unequal societies. In the north of the country, 72% of people live in poverty, compared with 27% in the south and 35% in the Niger delta, according to the US Council on Foreign Relations.

Goldman added: "It's a very big and very diverse country which has a particular source of wealth that has benefited some areas more than others. The north has been left behind and is probably more impoverished now than at any time in the last 30 to 40 years."

For centuries, the region enjoyed the fruits of Islamic civilisation. Then in the early 19th century its sultanates succumbed to a jihad by Shehu Usuman Dan Fodio, who created a unified caliphate that was the biggest pre-colonial state in Africa, ruling swaths of what is now northern Nigeria, Niger and southern Cameroon. It had a strict interpretation of Islam and a culture of scholarship and poetry. The current jihad is an example of religious rebellion in northern Nigeria that is still manifest, according to the anthropologist Murray Last. "My argument is that today's dissidents, such as the notorious Boko Haram, are part of a tradition of dissidence; and that neither are they a new phenomenon nor will they be the last of their kind," he wrote last year.

Northern Nigeria did not escape the expansion of the British empire into Africa and was conquered in 1903. Since then, there has been resistance to western education, with many Muslim families refusing to send their children to government-run "western schools".

Shehu Sani, a human rights activist and author of Boko Haram: History, Ideas And Revolt, said: "The north fought the British colonisers because they thought they were bringing in western ideas and this would erode Islamic values and erode their culture. The southern part of Nigeria was relatively more receptive. We can say Boko Haram has historical roots in resistance to the west, but it is not a justification for wanton killings. They are condemned by the vast majority of Muslims."

The north-east remained a centre of Islamic learning for children from all over Nigeria and west Africa, Sani said. Its madrasas did not necessarily encourage extremism but did shape the founders of Boko Haram, who embraced the Qur'anic phrase: "Anyone who is not governed by what Allah has revealed is among the transgressors."

Some believe the trigger for the group's inception was a gubernatorial election campaign in Borno state, when an opposition candidate organised a militia known as Ecomog, after the east African intervention force deployed in Sierra Leone and Liberia in the 1990s. Following the election, the candidate disbanded Ecomog but did nothing to look after its members.

One of the militia's leaders, Mohammed Yusuf, was able to exploit the frustration and disappointment and blend it with an Islamist agenda that rejected the failings of secular government to form Jama'atu Ahlis Sunna Lidda'awati wal-Jihad, People Committed to the Propagation of the Prophet's Teachings and Jihad.

In the north-eastern city of Maiduguri, where the sect had its headquarters, it was dubbed Boko Haram. Loosely translated from the Hausa language, this means "western education is forbidden". Poverty, joblessness and despair at feeble and corrupt central governance made for fertile territory among disenchanted young men. Even climate change played a part: the drying of Lake Chad meant that fishing families were displaced and had to search for alternative livelihoods, not always successfully.

Chris Ngwodo, a Nigerian political analyst, told al-Jazeera: "The group's appeal is religious and resonates in the context of a weak state with severely weakened institutions. Its theatre of operations – the Sahel – features a perfect storm of sovereignty: deficient states, a young, economically frustrated population mired in poverty, nations with long histories of strife and the collapse of agrarian economies due to climate change.

"Boko Haram represents an alternative order to this matrix of dysfunction. It evidently aims to be to the Sahel what the Taliban was in Afghanistan and in Pakistan's tribal areas."

Like so many self-appointed rebels and revolutionaries, Yusuf was not poor. He was said to be well-educated and to drive a Mercedes. In an interview with the BBC, he set out the group's anti-science philosophy: "Prominent Islamic preachers have seen and understood that the present western-style education is mixed with issues that run contrary to our beliefs in Islam. Like rain. We believe it is a creation of God rather than an evaporation caused by the sun that condenses and becomes rain. Like saying the world is a sphere. If it runs contrary to the teachings of Allah, we reject it. We reject the theory of Darwinism."

Yusuf set up a religious complex, which included a mosque and an Islamic school that attracted many poor Muslim families. In 2009 Boko Haram attacked several police stations and other official buildings in Maiduguri. The Nigerian security forces hit back and more than 1,000 people died, not all of them Boko Haram supporters. Yusuf was captured and killed, his body shown on television. Boko Haram was finished.

But its fighters regrouped under a new leader. In 2010 it attacked a prison in Bauchi state, freeing hundreds of its supporters, and carried out deadly bombings in Jos and military barracks in the Nigerian capital, Abuja. Its main modus operandi was to deploy gunmen on motorbikes to kill police, politicians and other opponents. Since then, the waves of shootings and bombings have continued and, according to the Council on Foreign Relations, Boko Haram is responsible for nearly 3,800 deaths since May 2011. The group has sworn allegiance to al-Qaida and, Sani says, some of its members have fought in Somalia and Sudan, but a formal link "cannot be independently confirmed".

Last year the president, Goodluck Jonathan, declared a state of emergency in three north-eastern states, but critics say the official response has been counterproductive, with extrajudicial killings and the torture of suspected Boko Haram members acting as a recruiting sergeant for the group.

Professor Ishaq Akintola, director of Muslim Rights Concern, said: "Life is very hard there. It is difficult to move freely from one place to the other. With military checkpoints all over the place and the fear of Boko Haram attacking them any moment, people are between the devil and the deep blue sea. Sometimes they are holed up indoors for days. Hunger and starvation have enveloped the environment. While thousands have fled their homes, it is only the fear of the unknown that has stopped the rest from fleeing."

If anything, Boko Haram has intensified its operations of late, including an attack that saw 43 students shot and hacked to death and many girls kidnapped. In response, the government closed five schools considered to be in "high security risk areas".

Some Nigerians who feel let down by the government are taking the fight on themselves. Zakari Matazu, survivor of the double bombing in Maiduguri, belongs to a youth vigilante group in Borno state popularly known as the Civilian Joint Task Force (CJTF). "Now Boko Haram are attacking everywhere because they are strong – even stronger than the soldiers," he said. "I am a CJTF but I now know that Boko Haram can decide to attack and capture the town of Maiduguri any time. Everybody knows that. The federal government has abandoned us to be killed by Boko Haram. All the people in the villages have fled to Maiduguri, so if Boko Haram does not see people killed in the villages, they will come to the city."

Last month Boko Haram threatened to strike farther afield, with potentially catastrophic consequences for the economy. Its leader, Abubakar Shekau, threatened attacks on oil refineries in the mainly Christian south, saying in a video: "Niger delta, you are in trouble." but few analysts believe the group poses an existential threat to Nigeria.

Goldman said: "People talk a lot of nonsense about Boko Haram. They say it's about Muslim versus Christian, north versus south. That's not true. Thousands have been killed and almost all of them have been Muslims in the north-east. That's where the brunt of this insurgency has been felt."

Several government crackdowns have failed to quell it and it seems 2014 could be the bloodiest year yet. Sani believes that only a combination of short- and long-term approaches can work. "The government of Nigeria can apply the stick – they must step up intelligence and military action – but there should also be the carrot approach if they can use Islamic clerics in the north and the Boko Haram members in detention. If there is a synergy of this, there could be a breakthrough. The long-term solution has to do with addressing poverty and inequality. There needs to be a new economic model for the Sahel."

Those on the frontline are living in a parallel universe to the champagne parties in Nigeria's big cities. "We are in a state of war," Kashim Shettima, the governor of Borno state, said recently in a plea to the president. "Boko Haram are better armed and better motivated than our own troops. It is impossible for us to defeat the Boko Haram."