The main square in Ilopango is bustling with noise as the sun disappears behind the distant mountains. The queue for papusas, tortillas filled with beans and melted cheese, is growing as hymns ring out from the evangelical church.

Across the square two girls – cousins aged five and six – persuade me to join in their boisterous game of football, oblivious to the young couples trying to enjoy a quiet moment. Turn the clock back a year, and this typical San Salvador twilight scene could not have existed.

The town was a battleground for the country's two biggest street gangs, Mara Salvatrucha 13 (MS-13) and Calle 18, with gang graffiti marking territory street by street. Both gangs used the square to display dead bodies, so everybody knew who was responsible. This public show of violence instilled deep fear among the street vendors, shop owners and taxi drivers, who were all forced to pay into the gangs' extortion rackets. No one came out after dark. El Salvador was ranked the world's second most murderous country in 2011.

Then, in March 2012, the gangs unexpectedly announced a truce. Leaders from both sides promised to stop the killings in return for more humane prison conditions and help in reintegrating members back into society. The Farabundo Martí National Liberation Front (FMLN) government, former leftwing guerrillas, risked the wrath of the gang-hating electorate by negotiating for peace. Within weeks the murder rate fell from 14 a day to five or six. Ilopango was one of the most dangerous municipalities of San Salvador, with 117 murders in 2011. The killings fell dramatically to 62 in 2012, and improbably it became the first area declared free from gang violence.

Marvin Gonzales, 29, a spokesman of MS-13 in Ilopango, is one of 30 gang members operating a chicken farm – a showcase truce project attracting international attention for the city's mayor. He tells me about choosing MS-13 over his family at the age of 12; this farm is his first legal job. "I served 10 years in hell for killing a boy whose name I didn't even know over territory. When my niece's father was sentenced to 136 years for triple murder, I knew this had to stop. I lost six friends to gang violence, I can't do it any more."

It's a steep, sweltering descent from the road to the farm, which overlooks El Salvador's largest lake. Though once a chicken farm, for the past 16 years this land served as a rubbish dump. Alex Renderos, 26, screws up his face as he recalls the mountain of stinking rubbish. "That's all we did for three months, it was disgusting, the insects were huge."

He was jailed for three years when his girlfriend was pregnant for possession of a 9mm gun. "I didn't see my daughter until she was three – which hurt here," he says, patting his heart. "That's why we are motivated to make this work."

Grinning, Gonzales hands me an envelope he's been carrying in his back pocket. His girlfriend is pregnant and he wants to show off the 12-week ultrasound scan. "I spent the best years of my life in jail, 100 men in a 20-by-15-metre room. Life for us pandilleros [gang members] is short: we end up in jail or dead. I don't want that for my child."

A couple of days later we drive to the flagship Calle 18 project – a tiny bakery down a side street that is covered in gang graffiti. The ovens are full and young boys are speeding off on bicycles loaded with bread rolls, honking their horns to attract customers. The senior gangsters aren't here, delayed at another meeting at the mayor's office with government officials, MS-13, and Father Toño Rodríguez – a controversial priest who recently joined the negotiating table.

Javier García, 22, has been working in the bakery for four months, earning $5 a day. He was kicked out of school at 14 and, like many teenagers, saw the gang as a laugh, better than being at home. "When I'm in the bakery I am not on the streets, so that's good for everyone," he says while lifting from the oven trays of pan francés – salty bread rolls that Salvadoreans eat with breakfast and dinner. "Maybe I will want to leave one day, but right now I still feel very strongly in my heart for my gang, we look after each other like the three musketeers."

García says his friend Kevin was killed by MS-13 only three months ago after stepping into their territory while selling avocados. "I can never imagine MS-13 as my friends, never. If I saw one now, I would hit him, for Kevin. For this truce to work, we have to stay apart."

The mayor, Salvador Ruano, came to power in Ilopango last May, after a narrow win for the rightwing National Republican Alliance, known by the acronym Arena in Spanish, in a FMLN stronghold – with the party punished for years of infighting by a record low turnout. Despite his squat physique, Ruano cuts an imposing figure as he bellows at me as if addressing a large crowd. "I promised both groups I would help them earn an honest living, as long as they kept their side of the compromise, and now I have them around the same table talking peace and people are definitely safer."

He dismisses critics as "short-sighted people with nothing to offer".

"The problem of violence here is decades old and there is no magic formula, no manual, no perfect solution… I cannot say Ilopango is free from violence, but after one year in power we are in transition from violence to peace. Those who criticise maybe have something to lose from peace."

Ilopango's square has a bloody past. During the 12-year civil war, it was from the airbase around the corner that the feared Atlacatl Battalion, a US–trained special forces unit, was sent in helicopters to massacre civilians in "red zones" such as El Mozote and El Calabozo.

Members of the urban guerrilla movement were captured and tortured and bodies were dumped in the square, says David Munguía, the director of FMLN Ilopango, whose office is adorned with pictures of the "martyrs".

"After one year, all we have is a bakery and a chicken farm that employs a handful of people, this cannot sustain [the] truce," Munguía says. "There are no social programmes or prevention projects and, OK, there are fewer murders, but delinquency, extortion and kidnappings are up.

"The Arena party doesn't support the truce because there is no money in peace." A recent news report valued the private security industry at £400m.

The fragile truce has powerful religious and political opponents. Norman Quijano, the mayor of San Salvador and Arena's 2014 presidential candidate, is pledging to get tough on crime and stop negotiating with criminals.

El Salvador has the most overcrowded prisons in the world, with 26,000 inmates kettled in prisons built for 8,400 – a direct result of the 2004 Arena Mano Duro or iron-fist policy. But Quijano's claims that murderous gangsters are now living it up with plasma-screen TVs hits a nerve, and his popularity is growing. One taxi driver tells me that he still pays $8 a week to Calle 18 – a quarter of what he earns – and wants them all locked up for life. Mauricio Funes, the president of El Salvador, promises to tackle extortion next, but the police, narcos and other organised criminals are all at it – so that is easier said than done.

Jeanne Rikkers, from the human rights and prison reform NGO Fespad, says: "Despite the US roots, the gangs were an El Salvadorean response to an El Salvadorean reality, flourishing in a perfect storm. State obligations to rehabilitate, reintegrate, improve prisons, reform education and health, need to be done transparently with access for all, not just negotiated for some. A promise to stop killing cannot solve underlying complex social problems."

Mejicanos, a suburb of San Salvador, will become the tenth area declared free from gang violence on 20 June, the third anniversary of the notorious Calle 18 arson attack in which 17 bus passengers died. Father Toño, who moved here from Madrid 14 years ago, chats over coffee while his guards – whose presence is the result of death threats from drug traffickers – loiter outside. "The gangs are the product of the unresolved causes of the civil war: huge inequalities in wealth, democracy and land ownership.

"The first step is to stop this war. The truce is not the solution, but without it there is no solution. This process could see the gangs becoming cultural organisations in 10 years. Or, if Arena comes into power with more iron-fist policies and support for dirty businesses like arms and security, the recruitment drive will start again."