Two Nato rockets aimed at Taliban insurgents in Helmand missed their target today, killing 12 civilians sheltering in their home and dealing a sharp blow to hopes that civilian casualties would be avoided in the largest western-led operation of the nine-year Afghan war.

The incident occurred in Nad Ali, an insurgent-infested area where British troops are operating. A Ministry of Defence spokeswoman said the rockets, which were fired by a sophisticated missile system, were a "US responsibility".

News of the deaths overshadowed cautiously optimistic reports from US commanders in nearby Marjah, a major hub of insurgents and drug smugglers, where marines and Afghan soldiers pushed deep into a labyrinth of mud-walled compounds surrounded by landmines and booby traps.

The soldiers went carefully, preceded by explosives teams and sniffer dogs. Taliban snipers holed up in farmhouses offered sporadic, and sometimes sustained, resistance. One group came under fire moments after they hoisted an Afghan flag over a newly captured compound.

Elsewhere, gunfire forced the operation's American commander, Brigadier General Larry Nicholson, to take cover behind an earthen bank. "The fire we just took reflects how I think this will go, small pockets of sporadic fighting by small groups of very mobile individuals," he told an embedded AP reporter.

Operation Moshtarak (meaning "together") involves 15,000 troops, mostly US, British and Afghan. The first US marines arrived in Marjah by helicopter before dawn on Saturday morning, while British forces are sweeping through Nad Ali.

The Taliban, estimated at 400 to 1,000 fighters, have heavily mined the area, and Nicholson predicted it could take 30 days to clear out all militants and explosives, although he was hopeful the job could be completed sooner.

Nato heavily telegraphed its intention to invade Marjah and Nad Ali, hoping to minimise civilian casualties. A Nato statement said the rockets that destroyed a house in Nad Ali landed 300 metres off target. They had been fired by a High Mobility Artillery Rocket System – a system that, according to several defence websites, is used principally by US marines.

"We deeply regret this tragic loss of life," said General Stanley McChrystal, the top commander in Afghanistan. "It's regrettable that in the course of our joint efforts, innocent lives were lost."

The deaths ratcheted up tensions with the Afghan president, Hamid Karzai, who ordered an immediate investigation. High rates of casualties help Taliban recruitment in war-torn areas and stir public anger that has eroded support for the fragile government.

At a press conference in Kabul the defence minister, Abdul Rahim Wardak, said the goal of the operation was "not to kill insurgents" but to "expand the government's influence and protect the civilian population".

Details of civilian casualties were sketchy tonight. A spokesman for the Red Cross said medics at a first aid post in Marjah had treated 30 people. He could not give a breakdown between civilian and military casualties.

Operation Moshtarak must succeed not only on the battlefield but in the follow-through by Afghan civilian and security forces. Kabul has promised to deploy 1,000 paramilitary police to Marjah in the coming days while the Nato civilian chief in Afghanistan, Mark Sedwill, said resource-laden "district development teams" were ready to deploy.

In London Bob Ainsworth, the defence secretary, told the BBC the "real test over the coming months" would be in "winning over some elements that have formally been opposed to us, that have been shooting at our troops".

The army said its air and ground assault – involving up to 1,500 Coldstream and Grenadier Guards, as well as troops from the Household Cavalry and the Royal Welsh regiment and nearly equal numbers of Afghan soldiers – had still not encountered major opposition. A spokesman said the Taliban standard had been replaced by an Afghan flag over a village in the Chah triangle.

The Ministry of Defence named a soldier from the 1st Battalion Grenadier Guards killed in an explosion on Saturday as Lance Sergeant Dave Greenhalgh, 25, from Ilkeston, Derbyshire. One US marine has also been killed.

Another British soldier was killed in an explosion yesterday while on foot patrol to the north-east of Sangin. The death of the soldier, from 6 Rifles, serving as part of the 3 Rifles Battle Group, was unconnected with Operation Moshtarak.

The army spokesman said the next stage – "hot stabilisation" – would begin soon. Reconstruction equipment being brought in includes box sets for 20 schools, solar lights, mosque broadcast sets and replacement culvert pipes. Recruitment of working parties would begin within days, with cash being made available for up to 37 teams of local labourers to start repairing roads.

Victory in Marjah would be a publicity coup for Barack Obama, who is struggling to bolster flagging US opinion. But many fighters are believed to have fled south to bases inside Pakistan or north into mountainous Urzugan province.

One Afghan official stressed that Marjah was just one of more than 700 districts in a country where broad swathes remain under Taliban influence. In a reminder of that threat, the Taliban released a video today of two French journalists kidnapped in December. On it, the two journalists plead for their release and urge the French president, Nicolas Sarkozy, to negotiate quickly with their captors.