The Somalia-based Islamist group known as al-Shabaab will emerge stronger and more unified after its terrorist attack in Nairobi, and could provide other extreme groups with an example to follow, counter-insurgency analysts warned on Tuesday.

Al-Shabaab's message is that it is "down but not out", it is "losing territory but not people", said David Kilcullen, a former adviser to David Petraeus, then US commander in Iraq, and of Nato forces in Afghanistan.

The purpose of the Westgate mall attack was to send Kenyans a message – it was the consequence of sending troops to Somalia, said Kilcullen, who was in London to launch his book Out of the Mountains: The Coming Age of the Urban Guerrilla.

He suggested that al-Shabaab wanted to provoke over-reaction by the Kenyan security forces and thereby gain more support, especially in Eastleigh, the eastern district of Nairobi where most of the 250,000 Somalis in the Kenyan capital live. Many in that area are already being radicalised, analysts say.

Kilcullen said al-Shabaab, which announced last year that it had allied itself with al-Qaida, was supported by about 5,000 fighters, many of them Kenyans, Tanzanians and Ugandans, but also there were about 40 westerners, most of whom were American with a few British.

British security sources estimate that 50 or so Britons have joined al-Shabaab in Somalia. But that figure is believed to have fallen. British counter-intelligence officials say Syria is now the "jihadist destination of choice".

With other analysts, Kilcullen predicted that there would be further attacks in east African countries, including Uganda, which had sent troops to fight al-Shabaab in Somalia as part of an African Union force, and Ethiopia, which had deployed troops across the border in Somalia.

"[Nairobi] was clearly a successful operation which required training and audacity, and will boost the organisation," said Raffaello Pantucci, senior research fellow at the Royal United Services Institute.

But he said it would be surprising, given the effort and individuals required, if al-Shabaab mounted a similar attack, at least in the immediate future. More likely was a series of low-level attacks, Pantucci said.

The US director of national intelligence, James Clapper, told the US Congress this year that al-Shabaab would probably "remain focused on local and regional challenges" and "continue to plot attacks designed to weaken regional adversaries, including targeting US and western interests in east Africa".

Kilcullen said that one advantage of al-Shabaab fighters was that they did not need a strong central command and control structure. They could operate in self-contained tactical cells, he said.

He also compared al-Shabaab with the Taliban in Afghanistan; both groups had benefited from over-reaction by security forces and both administered an "efficient" system of rules guided by sharia law.

Kilcullen also compared the attack on the upmarket Nairobi shopping mall with the 2008 attack by terrorists in Mumbai who targeted an area with a luxury hotel also frequented by westerners. That incident caused concern for MI5 as Britain's domestic security service contemplated such an attack in the UK.

"The lesson from this sort of attack," said Pantucci, referring to Nairobi and the perspective of an extremist group, "is that it can be very effective."

The hardline reclusive Ahmed Abdi Godane, also known as Mukhtar Abu Zubair, recently emerged as al-Shabaab's leader amid infighting and defeats of al-Shabaab fighters in Somalia.

Godane's tactics were considered too crude by Osama bin Laden, according to a declassified document published by the Combating Terrorism Centre at West Point, the US military academy.

On Monday, Stig Hansen, a Norwegian expert on al-Shabaab told the US broadcaster ABC that the African Union-led military offensive in Somalia has been successful in targeting al-Shabaab, which was now trying to establish networks outside the country.

"They are in decline in Somalia. That might push them to use more terrorist strategies. That's one scenario. And they haven't been in decline in the rest of east Africa. We can see that there's a nucleus of some kind of shape of network inside Kenya and maybe traces of the same thing within Tanzania," Hansen said.

British special forces, and officers from the Metropolitan police and MI6, are based in Nairobi advising and training Kenyan security forces.