The search for a government – any kind of government – to bring order to Somalia is growing increasingly desperate as warring Islamist factions, tribal clans and bandit gangs exploit a power vacuum created by this week's Ethiopian troop withdrawal. Just when it seemed the plight of Somali civilians could not get any worse, it did. Aid workers and human rights groups are not mincing words: catastrophe is just around the corner.

Hopes of staving off complete political collapse are pinned on UN-brokered talks in Djibouti on forming a national unity government. The idea is to bring together the discredited, exiled, western-backed transitional federal government (TFG) and the moderate Islamist opposition, the Alliance for the Re-Liberation of Somalia (ARS). The theory is that an expanded, more inclusive parliament will elect a new president, possibly the ARS leader Sheikh Sharif Ahmed, with power to reunite the country.

But facts on the ground are changing faster, and moving quicker, than diplomacy in Djibouti. Monday saw the fall of Baidoa, the TFG's last main base inside Somalia, to militants of the al-Shabaab Islamist movement. A plethora of groups and factions, some not heard of before, have reportedly seized, or are fighting each other for control of, other key towns in the centre of the country. The capital, Mogadishu, is in effect a lawless battlefield.

And since, in the short term at least, any new government will lack effective, reliable security forces (a recurring TFG problem), it is unclear how it could regain a foothold inside the country, let alone re-unite and rule it.

Even if there is a deal in Djibouti, al-Shabaab and an ARS splinter group based in Eritrea have already rejected the process as illegitimate. The Islamist hardliners say they have achieved their first aim: the departure of the Ethiopians. Next they want to eradicate other foreign influences, impose sharia law and, according to Sheikh Hassan Dahir Aweys, former leader of the Islamic courts movement, build an Islamic republic on the ruins of western conspiracies.

Regional analysts say this is the very nightmare scenario that the US, Britain and other EU countries have long struggled, though not very energetically, to avoid. For years after 1991, when the last functioning national government collapsed, they mostly sought to contain and quarantine Somalia. More recently they were content to let Ethiopia take the strain and the opprobrium, while US forces in Djibouti confined themselves to hunting al-Qaida suspects.

But now, as the country faces disintegration, the west's failure to support the under-resourced AU peacekeeping mission, Amisom, its effective refusal to deploy a UN force, its long-running efforts to gloss over TFG weakness and corruption and its lack of a cohesive overall policy could combine to create an epic policy disaster.

It's no mystery who will pay the highest and most immediate price. "The human rights and humanitarian catastrophe facing Somalia today threatens the lives and livelihoods of millions on a scale not witnessed since the early 1990s," Human Rights Watch's latest report warns. UN agencies say 3.25 million Somalis are already dependent on food aid; 1.3 million are internally displaced, including two-thirds of the population of Mogadishu. Twenty-five per cent of the total population is suffering from acute malnutrition.

Beset by conflict and drought, thousands more are fleeing each month in all directions – to north-eastern Kenya (already home to 220,000 Somalis), Ethiopia, Eritrea and, risking the perilous passage across the Gulf of Aden, to Yemen. This exodus is likely to grow significantly if the political impasse and related insecurity intensifies.

Even if the outside world suddenly wanted to do more on the ground, the inherent difficulties have become formidable after years of relative neglect. The World Food Programme said earlier this month that it might have to suspend food distribution after two of its employees were murdered.

Andrea Pattison of Oxfam Novib said fighting and lawlessness made it impossible for western aid workers to function safely in Somalia. "The biggest problem for Oxfam and other agencies is that we can't really access these people. Nearly 40 aid workers have been killed in the past year. There have been countless abductions and at least 150 of what are called security-related incidents."

All the same, Oxfam is providing water pumping and sanitation services in the Afgooye corridor, where many of Mogadishu's residents are camped out, and helping distribute 60,000 hot meals each day in the capital itself. But much more help was needed, Pattison said. "International action to end [Somali] piracy came very fast. Now it's time to show the same urgency about alleviating the suffering of millions of people."