Hundreds of desperate Tutsis sought refuge on the first day of the genocide at a school where 90 UN troops were under the command of Captain Luc Lemaire. Here, they were surely safe from the Hutus and their machetes.

The UN flag flew over the school. The Belgian peacekeepers were armed with a machine gun, planted at the entrance. These soldiers were the world’s army. The Tutsis could not imagine they would stand by while people were slaughtered.

The 39-year-old captain feared otherwise. The UN in New York had ignored warnings that the genocide was being planned and the security council was pulling out peacekeepers in response to the mass killing.

Within days, the UN command decided there was more important work for Lemaire and his men than protecting Tutsis. The peacekeepers were ordered to abandon the school in order to escort foreigners to the airport and out of Rwanda.

Lemaire’s contingent kept a sporadic video diary of the wretched events. At one point a shaky camera captures a sandbagged machine-gun post and a pickup truck of Hutu militiamen sweeping by.

The video then focuses on the soldiers’ log. It reads: “New York didn’t agree to change the rules of engagement.” The peacekeepers were not to be permitted to save Tutsis. The camera’s microphone picks up an anonymous voice: “There are killings and New York doesn’t give a damn.”

The betrayal of the Tutsis in Rwanda was a low point for UN peacekeeping but not an isolated one. A year later, Dutch peacekeepers failed to stop the massacre of 8,000 Muslim men in Srebrenica, a supposedly UN “safe area”, the most notorious mass killing by the Serbs in Bosnia.

General Ratko Mladić of the Bosnian Serb army, left, drinks with Dutch colonel Ton Karremans, second right, in the Bosnian village of Potočari, on 12 July 1995, the second day of the massacre. Photograph: Anonymous/AP

Not long before, there had been the debacle in Somalia where a US-led UN humanitarian operation turned into a bloody conflict against a powerful warlord. By then Angola was already back at war after its UN peacekeeping mission collapsed amid accusations it contributed to the breakdown of peace.

Other disasters – the Democratic Republic of the Congo (DRC), Sierra Leone – were still on the horizon even as the UN peacekeeping department’s budget doubled and doubled again with growing numbers of missions.

70 years and half a trillion dollars later: what has the UN achieved?

Read more

There were good reasons to question the point of UN peacekeeping in the 1990s and to wonder if it was not costing as many lives as it was protecting by offering an illusion of security. The murdered Tutsis at the school might have stood a better chance if they had fled to Uganda.

Two decades later, those responsible for peacekeeping – UN officials, countries on the security council assigning missions, militaries contributing the forces in blue helmets – say the lessons of the 1990s have been learned. Protection of civilian lives is now a priority. UN soldiers no longer have their hands tied in using force to defend the innocent. Perhaps most important, the failures of two decades ago stripped the UN of the illusion that an election and a few soldiers in blue helmets is enough to sustain peace.

But even as peacekeeping has ballooned to become by far the most expensive of UN departments (in 2015 it will cost nearly $9bn to keep 120,000 blue helmet soldiers and policemen deployed in 16 countries from Mali to Cyprus and Haiti, compared with just $500m at the end of the cold war), it is dogged by challenges.

The politics of the UN security council continues to hamstring action over some conflicts, such as Syria and Darfur. Other missions drag on indefinitely, such as in the DRC and Haiti. There is friction between the UN and some governments over who has ultimate command over peacekeeping troops, which has resulted in conflicting orders with disastrous consequences. Then there are spreading conflicts against armed Islamist groups in central and west Africa.

“In the last 15 years, the number of missions, number of crises have drastically multiplied,” said Ban Ki-moon, the UN secretary general. “In addition to using military and police forces, we have been deploying a lot of special political missions. This is preventative diplomacy. If we can prevent at the time of symptoms of a crisis, we can save a lot of human lives and human resources.”

Jack Christofides, who is on leave from the UN as director of peacekeeping operations for central and west Africa, described peacekeeping as “extremely taxing at the moment”.

“We’ve got more troops, we’ve got a bigger budget, we deployed in all sorts of very very difficult places, much more difficult than we’ve ever been, and we’re stretched, we’re really stretched,” he said.

“If you think of the old deployments in places like Lebanon and Bosnia, there is a certain infrastructure you could use and work with. The troops coming were generally from countries that had the means to launch expeditions. Today, when you’re talking about northern Mali and central Africa, you have both extremely dangerous conditions and geostrategic locations which are very difficult to get to.”

And there are added demands as the US presses for peacekeepers to take on a more aggressive role, particularly against armed Islamist groups in Africa. President Barack Obama is jointly hosting a summit with Ban in New York later this month to seek commitments to strengthen peacekeeping with better trained troops, equipment and intelligence resources, to the alarm of some of the countries that provide substantial contingents of troops in blue helmets.

Indian soldiers gather before their departure to Sudan on a UN peacekeeping mission. Photograph: Raveendran/AFP/Getty Images

India, which has sent more soldiers on UN missions than any other country – 180,000 on 49 missions – is openly challenging the move towards what some see as mostly rich and powerful countries on the security council sending the poor to fight and die.

“India is completely committed to peacekeeping provided peacekeeping is what we know it to be,” said India’s ambassador to the UN, Asoke Kumar Mukerji. “The soldiers in the blue helmets, under the blue flag, are impartial. They are not supposed to be partisan. If somebody wants soldiers to go in and fight they should hire mercenaries, not take UN soldiers.”

UN peacekeeping came into its own with the collapse of the Soviet Union in 1991 and the diminishing of ideological rivalries that had dominated international politics since the second world war.

Early successes in Cambodia, Namibia, Mozambique and El Salvador generated an overconfidence in the ability of UN soldiers to keep the peace. Each of those countries had an accord that former warring parties wanted to maintain.

The UN learned the hard way in Angola, Rwanda and Bosnia that where the UN wants peace more than those in conflict, then the illusion of peacekeeping can perpetuate instability and cost lives.

Out of the disasters of the 1990s came a new model including an ethos of “responsibility to protect”. No longer would UN forces stand idly by while innocent people were murdered. That has helped drive a rapid expansion of peacekeeping to become the UN’s single largest operation.

In 1991, the year the Soviet Union collapsed, the peacekeeping budget was just $490m. Over the past decade it has leapt from $3.8bn to about $9bn as the size and length of missions has expanded. More than 100,000 UN peacekeepers have served in Haiti since 2004 and close to 250,000 been deployed to the DRC over the past 15 years. There is little prospect of either mission ending soon.

Today, UN troop deployments in the DRC, Sudan and Darfur each cost more than $1bn a year, with Mali and Central African Republic not far behind. The most expensive outside Africa is Haiti, with a budget of $500m.

The emphasis on Africa is in part a reflection of the enduring nature of some conflicts on the continent, but also the politics of the security council. The UN might have had large peacekeeping missions in Syria and Ukraine, were it not for Russian objections.

As the scale of peacekeeping has grown so have the questions about its role. Most particularly, are peacekeepers there to monitor peace or to enforce it? It’s a source of increasing friction between the major powers on the security council that approve UN intervention and some of the countries which provide the troops.

A soldier from the Armed Forces of the Democratic Republic of the Congo sits on a personnel carrier next to a UN peacekeeper outside Goma. Photograph: AFP/Getty Images

In 2000, British forces landed in Sierra Leone after UN peacekeepers stood aside or fled an advance on the country’s capital, Freetown, by a notoriously brutal rebel group, the Revolutionary United Front (RUF). Several hundred peacekeepers surrendered to the rebels.

A British general, David Richards, was sent in by London to evacuate foreign citizens. Confronted with the RUF’s horrific record and the imminent killing of large numbers of civilians, he defied his instructions and used UK troops to spearhead a counter attack.

Richards, who went on to become Britain’s chief of the defence staff, said he was “staggered” to find that the Indian UN force commander, Major General Vijay Jetley, interpreted his mandate as that of a neutral intermediary. In Richards’ view the UN should have been siding with the elected government against rebels breaking a peace agreement.

“I had a real argument with Major General Jetley about this. The Indian hierarchy were very reluctant to lose a single person on a UN operation,” said Richards. “They were very reluctant to fight and that permeated quite a lot of the other contingents as well, to the point where I remember going in to the UN headquarters on my first day there in May 2000 and finding Jetley. I said General, we’ve got to stop the RUF, you’ve got to tell your people to fight. At least hold their positions. He was very reluctant to do it.”

In an internal UN report, Jetley accused Garba and other senior Nigerians of being more interested in smuggling diamonds than keeping the peace. Nigeria’s military responded by accusing the Indian general of “trying to justify his ineptitude, inaction and inefficiency in the leadership of a multinational force”.

An Indian UN soldier watches an RAF Chinook helicopter land at the UN headquarters in Freetown, Sierra Leone. Photograph: Peter Macdiarmid/AFP/Getty Images

Such differences of opinion over the role of UN peacekeepers were replicated 12 years later in the Democratic Republic of the Congo as rebels advanced on the eastern town of Goma. The Indian commander of part of the largest peacekeeping force in the world ignored orders from UN officials to defend the town and called the Indian defence ministry in New Delhi to ask what he should do. He was told not to resist. The rebels seized Goma to the anger of Ban Ki-Moon, who regarded it as a “personal humiliation”, according to a senior UN official.

By then members of the wider UN mission in the DRC had come to look like bystanders to mass killing, rape and terror. Richard Gowan, until recently research director at the Centre on International Cooperation, a thinktank in New York that works closely with the UN on peacekeeping, said Indian forces in Sierra Leone and the DRC were taking orders from the defence ministry in New Delhi, not the UN commanders on the ground.

“It’s very clear that there are directions from New Delhi to avoid all risk, and that there are parallel lines of command between New Delhi and their guys in the field. That’s true of a lot of countries. A lot of field officers, if they get any order to take a significant risk in a peace operation, they’ll immediately phone home. That’s making it almost impossible for some force commanders to get anything done because they have no true control over their own units,” he said.

Most western nations will not put boots on the ground as frontline peacekeepers and so the UN is dependent on the goodwill of those countries prepared to deploy troops such as India, Bangladesh, Rwanda and Nigeria, making it hard to assert its authority.

The UN lost confidence in Jetley in Sierra Leone but when the then secretary general, Kofi Annan, tried to remove him, the Indian government threatened to pull out all of its forces. Annan’s successor, Ban, ran into the same threats from New Delhi when he tried to remove the Indian UN commander after the Goma debacle.

Rwanda and Srebrenica prompted a greater emphasis on protection of civilian lives, and involvement in rebuilding failed states instead of negotiating peace treaties, standing back and hoping it all worked out. Gowan said that while the “responsibility to protect” ethos was laudable it had also led to ill-thought-out missions.

“Sometimes in peacekeeping, the security council says ‘Good God we need to have a mission’ and that mission is stuck together,” he said.

A convoy of the UN mission in Darfur crosses through a mud track in the southern village of Kashalongo, south of Nyala. Photograph: AFP/Getty Images

Gowan pointed to Darfur, where he said the UN intervention came as a politically panicked response to public pressure over the mass killings by the government-allied Janjaweed militia.

“For political reasons the US and the UK insisted on having the mission. I’ve spoken to UK officials since who’ve said: ‘We had no real idea what this mission was meant to do. We were just under such public pressure to come up with an answer and the answer was peacekeeping.’ That was definitely true for the Bush administration as well,” he said.

“Darfur has been a quagmire from the beginning. The frank reality is no one believes that the mission is working but no one dares pull it out because they fear the moment it goes there will be an even greater spike in violence and the security council will be held responsible. It’s become a slow burning disaster.”

Philippe Bolopion, the UN director at Human Rights Watch, said the principal reason for the failure of the Darfur mission, a hybrid operation with African Union forces, was deadlock on the security council.

“When permanent members of the security council can’t agree to stand up to an abusive government such as Sudan’s, and you have weak peacekeeping troops on the ground, it’s almost a perfect storm where peacekeepers are not going to protect civilians properly,” he said.

Members of the security council routinely prioritise national interests over the needs of the peacekeeping missions

“Permanent members of the security council routinely prioritise their national interest over the needs of the UN peacekeeping missions they have mandated, as a result often undermining them. Russia and China have done this by opposing more sanctions against the Sudanese government even when it pushes peacekeepers around in Darfur.”

The responsibility to protect has had a more positive impact elsewhere. Peacekeepers in South Sudan turned their bases into de facto refugee camps protecting tens of thousands of people. That would have been unlikely 20 years ago.

But the ethos has been severely challenged by the fallout from the security council mandate for military intervention in the 2011 Libyan revolution to protect civilians in Benghazi. Russia accused Nato of abusing it to support the overthrow of Muammar Gaddafi. That in turn has complicated UN action on Syria.

A UN flag flutters amid the rubble of the Lebanese-Israeli border town of Maroun al-Ras. Photograph: Patrick Baz/AFP/Getty Images

Finding common ground on sub-Saharan Africa has generally been less difficult because of fewer strategic rivalries. The debacle in Goma prompted the UN to put together a fighting force of soldiers prepared to go into combat. South Africa, Tanzania and Malawi volunteered to send troops to join the Force Intervention Brigade in part because they were weary of the persistent instability in the region.

“We were all embarrassed, humiliated at the end of 2012 when Goma fell,” said Christofides, who oversaw UN peacekeeping operations in the DRC from 2011 until earlier this year and was an architect of the intervention brigade. “Everybody has that image of peacekeepers sitting on top of an APC [armoured personnel carrier] and this group of ruffians walking into Goma. That was a low moment for everybody.

“When we recommended the mandate of the intervention brigade, the security council did not need much persuading. You have some very far-reaching language in the mandate about targeted offensive operations. That’s not language you see in resolutions of the UN. Neutralising armed groups, preventing their expansion, disarming them.”

The UN also deployed drones for the first time to help monitor the movements of the rebels and to verify Rwanda’s denials that it was supplying the insurgents. The drones are not armed although that appears the next likely step, if not in the DRC then in some other conflict where the UN is involved, given their increasing use in the Middle East by the US and its allies.

The intervention brigade has been successful in changing the dynamic in eastern Congo by pushing back the rebels and providing a degree of stability to begin the reconstruction of functioning systems of government, such as the courts – although regional politics has played a large part too. But the operation has also deeply alarmed countries such as India, which fears it sets an example for the future of peacekeeping.

Christofides doubts that the force intervention brigade provides a model for other peacekeeping missions. Senior UN officials, some of whom were strongly opposed to the creation of the brigade but now judge it a success, are deeply wary of the UN taking on a similar role in other conflicts. The DRC, they say, was a unique situation.

But even if the force intervention brigade is not replicated, it fits with a wider agenda by the major western powers to remake UN peacekeeping as a partner in military campaigns. For them, the lesson of British intervention in Sierra Leone or the recent French deployments in Mali and Central African Republic against armed Islamist groups is that better trained and equipped forces launch the initial assault and then the UN comes in as backup. But for that to happen, UN peacekeeping has to up its game.

Obama is leading the push with his New York summit on 28 September to seek pledges of troops, equipment and intelligence resources to a revamped peacekeeping operation.

Washington regards peacekeeping as a national security issue and a “good bargain”, as one official put it. The US picks up the tab for a quarter of the UN’s $9bn peacekeeping budget, but it is a fraction of the cost of the Pentagon or invading Iraq.

Moreover, if other countries are sending their forces then the US does not have to risk the lives of American soldiers. Previously, the US mostly regarded peacekeeping in Africa, in particular, as a humanitarian issue. Now, given the nature of the conflicts in Mali, Nigeria and Central African Republic, Washington views it as strategic.

But while the US wants more assertive peacekeeping, it does not want to send its soldiers to fight. Obama is pressing more developed countries, in Europe, Asia and Latin America, to make greater commitments in the hope of improving the quality of UN forces. So is Ban.

“I’m now in the process of reaching out to developed countries. I’ve been speaking to many presidents and prime ministers to diversify the participation of peacekeeping forces,” he said.

“If we can diversify and have more countries, particularly some big countries, if they can send their own troops in addition to assets and financial support, that would be much appreciated.”

UN officials privately concede there is little chance of the US putting its forces under UN command. Congress would never stand for it. But Washington is joining the UN in pressing other developed countries to play a bigger part in peacekeeping.

The UK currently has fewer than 300 soldiers deployed on peacekeeping missions, mostly in Cyprus. “We’ve been having goodness knows how many discussions with British officials with a view to getting them more involved,” said Christofides. “We’re still drawing too many troops from a few parts of the world and not enough from other parts of the world. And I don’t just mean Africa versus the west, but other countries in Asia and Latin America that could contribute a bit more.”

Another senior UN official put it more bluntly: “Britain has a reputation for lecturing without contributing.”

Richards wants to see Britain make a greater contribution. “The Ministry of Defence and, I have to say sadly, the armed forces, don’t really see the UN as proper soldiering. This is a cultural ignorance that’s grown up over many years. The Americans share it writ large: UN ops is what second- and third-world nations do but proper armies, we pick and choose,” he said.

Richards said countries whose forces had experience in Afghanistan and Iraq could put that to good use in UN operations. “I think that’s a pretty good role for the British armed forces in this era when there’s so many nasty things going on in the world. The Americans really need to decide to do it as well,” he said. “I think at a low level, our young officers would love to do it. There is a higher level political/military reluctance and snootiness about it.”

India strongly opposes a move towards more forceful peacekeeping. “When it has been used as a tool to ensure that a peace agreement is observed so that peace building can take place, or as a tool to facilitate a political resolution, it works,” said Mukerji, the Indian ambassador to the UN.

“But if peacekeeping is to be seen as peace enforcement, then unfortunately we can’t see the UN charter allowing such a radical departure of the use of peacekeeping.

“Peacekeeping is not an end in itself. The end is political stability and peacekeeping is just a tool to bring about political stability. What’s happening now is the cart is being put before the horse. I think that’s a very unfortunate development.”

Gowan is sceptical for different reasons. “I think that we may be stumbling into an enormous strategic trap because if we have learned over the last decade that very highly capable Nato forces, US forces, actually can’t suppress Islamic extremist groups, why on earth do we think slightly strengthening UN missions is going to give us a tool that allows us to fight terrorists?” he said.