Fighters loyal to Muammar Gaddafi have seized back the town of Bani Walid and raised the late dictator's green flag, in a blow to Libya's struggling provisional government.

Reports said at least four people were killed during clashes between besieged forces loyal to the ruling National Transitional Council (NTC) and armed and well-organised supporters of Gaddafi. "They control the town now. They are roaming the town," one militia member was quoted as saying of the pro-Gaddafi fighters, according to Reuters.

Bani Walid, a former regime stronghold 110 miles south-east of Tripoli, was one of the last to succumb to pro-government forces after the capital fell in August. The latest clashes mark the most significant loyalist attack since Libya was officially "liberated" on 23 October. It appears further evidence of the NTC's weakness, incapacity and internal divisions ahead of supposed national elections later this year.

About 30 pro-government were reported to have been injured in exchanges of fire. Mahmud Warfelli, spokesman of Bani Walid local council, said 100-150 men armed with heavy weapons launched a carefully planned attack, swiftly overwhelming the town. Soldiers from the May 28 Brigade were pinned down in a compound and the attackers took up sniper positions in a mosque and a school, preventing the pro-government fighters from helping their wounded to safety.

Warfelli said the defenders' ammunition supplies had almost run out and he had had appealed to Tripoli and the defence ministry to send reinforcements. No help had come, he said. "We're out of the frying pan into the fire. We've been warning about this for the past two months," he said, according to AFP.

Units from the coastal city of Misrata, home to the most powerful of the government militias, scrambled armoured units towards Bani Walid with orders to block routes out of the town. Libya's defence minister, Osama Jweli, said he would not order army units to enter until he had established whether the fighting was indeed a pro-Gaddafi uprising or a battle between rival clans. "There is conflict at Bani Walid," Jweli told the Guardian. "For the moment we are waiting to assess the situation."

Bani Walid was the scene of prolonged fighting last November when pro-government forces entered the town searching for war crimes suspects and battled with local militias, leaving twelve fighters dead. Skirmishes between competing militias have been common in western Libya since the Gaddafi regime was toppled last October. But the recapture of Bani Walid is something new – not least because with Gaddafi dead and his son Saif al-Islam in custody, anti-government forces have no leadership figure around whom they can unite.

The green flag of Libya's ousted regime was reportedly flying again over many parts of the city. There were reports that the attackers were shouting the old regime slogan: "Allah, Muammar, Libya, that's it!" They were also carrying green flags.

The NTC is supposed to be paving the way for a new constitution and democratic elections, but it has been struggling to assert its political authority. On Sunday it was due to announce a new electoral law and the composition of an election commission. The announcement was delayed after protesters ransacked the NTC's offices in the eastern city of Benghazi, where Libya's revolution began almost year ago.

The protesters, made up of former rebels, had been demonstrating outside the building for weeks, unhappy at the lack of transparency within the NTC and the apparent appointment of ex-Gaddafi loyalists to the ruling body. They believe that opportunists have been allowed to join the government, with some saying the NTC represents western rather than Libyan interests.

Interim officials counter that it is impossible to sack all of the officials who worked for the Gaddafi regime, and say those involved in human rights abuses and other crimes will be prosecuted.

On Sunday the NTC's deputy leader, Abdul Hafez Ghoga, resigned after being manhandled at the protest.

Meanwhile, the international criminal court denied a claim by the Libyan justice minister, Ali Humaida Ashour, that it had agreed that the war crimes trial of Saif Al Islam would take place in Libya. The court said no decision had yet been taken.