The British soldier who went missing on Monday was stationed at a new base in the Nahr-e Saraj district of Helmand province, which recently came under sustained attack by insurgents.

He had been on sentry duty at Checkpoint Salaang on Sunday night and early Monday, and had just finished his shift.

It would most likely have been quiet at that time, but still warm: in Helmand at this time of year, the temperature rarely falls below 25C.

His colleagues from 4th Battalion the Royal Regiment of Scotland, and those from the Afghan army who are also at the heavily fortified compound had expected him to return to quarters. Instead he appears to have left the base on his own. Within hours stories were circulating about what happened to him next.

The Taliban claimed he had been captured by insurgents and killed in a skirmish as Nato forces tried to rescue him. Another report said he had been seen walking to the nearby Nahr-e Bughra canal, perhaps to go for a swim.

Without having complete clarity themselves, military officials in London were as certain as they could be that these stories were untrue.

But his disappearance was considered serious enough for David Cameron to cancel a trip to the British base at Lashkar Gah, the capital of Helmand, and for the full might of Nato forces – surveillance planes, helicopters and hundreds of extra troops – to flood into the area to seek him.

Sources in London said they were baffled by what had happened and were still trying to establish what had encouraged the soldier to leave his post without telling colleagues.

Just days ago, a patrol of Royal Marines from 45 Commando came under attack from insurgents hiding just 300 metres from the base, so security would have been at its highest level.

"He had been on 'Stag' [sentry duty] at a small vehicle checkpoint and had just finished his shift," said one source.

"And then, for whatever reason, he walked off. Instead of going back into the base where the other soldiers were, he headed out of the base – and that was the last that was seen of him.

"We are trying to work out exactly what has happened. There was no battle or fight or anything like that as far as we know."

The Ministry of Defence was told that the soldier was seen heading down the road away from the checkpoint by Afghan soldiers, who then reported him missing.

As the search got under way, the International Security and Assistance Force (Isaf) issued a brief notice stating that a soldier had "been listed as duty status whereabouts unknown in southern Afghanistan. There is an active search effort in progress."

Reporters travelling with the prime minister were told too, as the entourage was diverted – at Cameron's request – away from Helmand to the capital, Kabul.

The Taliban were quick to claim responsibility for the soldier's disappearance. Speaking to the Guardian, Qari Yousuf Ahmadi, a Taliban spokesman, gave one detailed account.

"Yesterday afternoon ... our mujahideen ambushed some soldiers and captured a British soldier. At 8pm, the foreign forces operation began to rescue him and continued until 3am.

"During that operation, the British soldier was killed, along with four to six other foreigners. Now the foreigners are saying that he was alone. But he was not alone.

"They are not brave enough to come even two steps out of their base alone. His body was left in the battlefield. I don't know where it is now." Ahmadi said he did not know how the soldier was killed.

The MoD said this account was not true. "There are always claims of responsibility for incidents," said one Whitehall official. "But we can only work with what we know. And all we know is that he has gone.

"At the moment, we cannot even be absolutely sure he is in the hands of the Taliban. There are criminal gangs operating in this area too. A huge search operation is under way. That has to be our focus right now."

The reports from Afghanistan also prompted the defence secretary, Liam Fox, to address MPs. "I recognise there will be many questions, but speculation on an issue of this nature is unhelpful. I would urge restraint from colleagues and the media and assure the house the United Kingdom and Isaf are taking all necessary and appropriate action." Two hours after that statement, officials confirmed that the soldier's body had been found and he had become the 375th British serviceman to die on duty in Afghanistan.

If the insurgents have claimed another victim, it will serve as a chilling reminder that the fighting in Afghanistan is far from over and the Taliban – despite the pounding they took over the winter months – are still capable and resilient.

Isaf commanders repeatedly warned that insurgents would seek publicity over the summer months with high-profile international targets, and so far they have been proved right.

Last week suicide bombers attacked the Intercontinental hotel in Kabul and before that targeted a hospital in Logar. Two months ago the Taliban orchestrated an audacious escape from Kandahar prison.

Monday's incident comes during a particularly difficult time for UK forces in Afghanistan. Eight British soldiers have died on duty in the past five weeks as the Taliban seeks to reassert itself in areas where it had apparently lost control.

Commanders have also been at odds with their political masters over whether British and American forces are being pulled out of the country too soon. This latest episode could not have been timed better from the insurgents' point of view. In just over a fortnight the "transition" process – Nato forces handing over control of parts of the country to the Afghan army and police – is due to begin.

The safest districts are to be in the first tranche of transition, and those currently under British control are included in this first wave. The improvement in security at Lashkar Gah is often cited as a British success story and the district is scheduled to be transferred to the Afghan authorities on 21 July.

Adopting the counterinsurgency doctrine that has become synonymous with US general David Petraeus, British commanders have taken a high-profile role in the city, spending time with locals and working with Afghan police and army officers to keep the peace.

But even here there has been trouble. In May the Taliban attacked a number of Afghan checkpoints – one rocket exploded near a police station and killed the daughter of a police commander. The compound of the governor, Gulab Mangal, was also hit.

The military would argue that the security situation is better, and there is some evidence to back the claim. In May 2010 British troops in Helmand had "contact" with insurgents 160 times in four weeks (15 in the first week of May, 30 in the second, 50 in the third and 65 in the fourth). In May 2011 the total was 96 "contacts" (11, 10, 45, 30). If the established pattern continues, the figures for June are likely to be higher.

But military officials know that impressions count and if insurgents appear able to target British troops, local people will question whether they are ready to put their security in the hands of the less experienced Afghans.

The total number of security incidents across the country may not reassure Afghans either, with the number of incidents barely changing year on year, despite the best efforts of the American "surge". Commanders argued that this lack of improvement was predictable as the Taliban attempted to regain lost influence and territory.

They have also insisted the surge should only be judged after two full years. But this is a luxury they no longer have. Barack Obama has ordered all 30,000 extra US soldiers sent to Afghanistan last year back home by autumn 2012. Cameron, meanwhile, is expected to announce that the British will withdraw another 500 troops next year, in addition to the 450 leaving this year.

Nato countries now have a blueprint to get out of the country and an end goal – by December 2014 they will be in Afghanistan for training and overseeing purposes only, not for fighting. The entire process is predicated on the belief that Afghanistan will be comparatively stable by then, with their security forces able to take over. But those are two very big assumptions in a country where Nato has learned not to assume anything.