It started one humid afternoon when a seven-year-old boy flicked something out of a schoolbus window at a teenager on García de Sena street. Some say it was a piece of popcorn, others a rolled up piece of paper. Whatever it was, the older boy took umbrage. He strode to the window and slapped the child, sending him home crying to El Cementerio, a few blocks up the hill.

The mother, aunts and sisters stomped down the hill and confronted the teenager, jabbed him, prompting his own female relatives to jab back. Insults flew. That night, shots were fired into El Cementerio. The neighbourhood did not see who was shooting but returned fire. And so began another war.

Eight months and multiple ambushes later seven youths from El Cementerio are dead, as are an uncertain number of their rivals down the hill. Richard Nuñez, the soft-spoken leader of the Cementerio youths, lifts his T-shirt to show fresh scars on his belly, back and right arm. "They got me when I was riding past the police station, shot me right up." He survived, but wonders if he will be as lucky the next time. "Things are pretty hot. This isn't over."

Over. A hopeful word. As if the violence has a destination, an end point. This is Venezuela, where more than 14,000 people were murdered last year, according to human rights groups. That is about three times bloodier than Iraq, which has a similar population. The government does not publish full statistics but says the official murder rate is 48 per 100,000 people, more than double South America's average. Some estimate the rate in Caracas to be as high as 140 per 100,000, making it one of the world's deadliest capitals. Hospital emergency wards overflow, especially at weekends, with bleeding, punctured casualties. Corpses stack up in morgues while grief-stricken relatives gather outside, noses cupped against the smell.

What makes this corner of South America, once best known for oil and beauty queens, a Hobbesian lottery? The short answer is gangs. Young men with guns drop bodies as they battle over turf and drugs in winding, rubbish-strewn streets. The catch-all description for them is malandros, supposedly feral thugs and ne'er-do-wells perpetually at war with themselves and the rest of society. They inhabit, Venezuelans tell you, the land "up there": hillside barrios. Malandros flit across television screens and newspapers as cadavers or hooded suspects paraded by police. Either way they are anonymous cyphers who do not speak, leaving their motivations, their world, incomprehensible to outsiders. A war over a piece of popcorn?

This is the story of one gang. Of its rise and fall and resurrection in a dusty, sun-baked slum, and of the reasons it does what it does. Some of the plots and characters make US crime dramas seem tame. There is the hitman who became a minister's bodyguard. The straight-A student suspected of black magic because no one can kill him. The mugger who found love while dodging police. The prison cannibal who found God. And the aristocratic rum merchant who proved an unlikely saviour. The narrative tilts between decay and hope, corruption and redemption.

Fifty years ago El Consejo was a sleepy farming village of 2,000 people ringed by sugar cane plantations. Today it is home to 50,000 people, none of whom farm, and whose brick-and-tin homes cling to steep slopes. It is a community marooned without jobs and proper housing by dysfunctional oil booms that stunted industry and agriculture.

Caracas is 60km east, at the end of a potholed motorway, but El Consejo feels like a ramshackle extension of it. Tucked into its concrete mazes, unmarked on most maps, is the two-block neighbourhood of El Cementerio, so named because it abuts a graveyard. Here, says Jimin Perez, a former police officer, is where you don't want to stop and ask directions. "The kids stick a gun in your face and steal your things. Then the adults dismantle your car for parts. If you make it out and go back with the police, no one has seen anything."

This is the fiefdom of the cemetery gang: two narrow roads lined with bleached houses from which eyes appraise all who enter and leave. Everybody knows each other, and many are related. Most men are gone – absconded, dead, jailed – leaving wives and widows as matriarchs to raise broods alone. The nearest school, Manuel Cipriano Perez, is so overcrowded its 1,117 students are rotated in two shifts. Often there is no electricity or air conditioning, so pupils slump in the tropical heat. The computer room is locked and empty. "Children don't have many recreation options," says Damaris Costa, the director. "They throw stones at their own school."

A five-minute walk down the hill from El Cementerio brings you to identical-looking streets, but this is the territory of Los Pelucos (The Wigs), the other gang in the "popcorn war". Walk 10 minutes west and you are in the territory of the 5 de Julio (Fifth of July) gang. It too is at war with El Cementerio but over a motorbike stolen in 2008. Bubbling under these disputes is competition to sell drugs, mostly cannabis and cocaine, to outsiders.

Richard Nuñez, leader of El Cementerio gang, shows the scar where he was shot. Photograph: Sean Smith for the Guardian

Wedged in by bigger rivals, El Cementerio has responded with tough leaders, none more so than Darwin Ospino, aka Pata Piche, or Rotten Foot. The nickname is ironic. Fastidious about deodorant and aftershave, Ospino is the neighbourhood's closest thing to a metrosexual. His fame, however, rests on reputedly having killed 26 people, a number that happens to match his current age. His first gun was a 765 revolver. "It felt like a trophy," he says, perched on his bed, freshly showered. "The first time I used it? A party. A gang showed up, made some trouble . . ." Ospino's voice trails off. He does not care to dwell on the details.

Alberto Vollmer, who owns a rum company that rehabilitates local gangsters, is more open. "Darwin was the hitman. He started at 16 and is still a legend around here. He's respected even by his enemies." Some stories are particularly chilling – dragging people from homes and families and shooting them in the face, killing a woman's husband, then killing her second husband after she remarried – but Ospino's manner is courteous and quiet, even diffident. "My papa abandoned us when I was small," he says, when prompted. Ambitions for the future? "Stay alive and be a good father." He has two young children but is separated from the mother.

Ospino apparently stopped killing in 2003 when, en masse, the Cementerio gang, exhausted from ever-present danger and stress, joined the rehab programme run by Vollmer's Santa Teresa rum company. Ospino, who inspired such fear that victims' relatives never pressed charges, was hired as a bodyguard by a government minister, Jesse Chacón. All went well for five years until he was accused of another killing, which he denies. Until the case is heard a court has confined Ospino to the Santa Teresa hacienda, which he patrols on a motorbike. "If he goes to prison he's a dead man, he's got too many enemies," says Vollmer.

Venezuela has no capital punishment but prison is often a death sentence. On average more than 420 inmates die violently each year, according to the Prison Observatory, a watchdog group. A system designed for 14,000 holds 38,000, most of them on remand. Guards control, at most, the perimeter, leaving inmates to fend for themselves. A September riot in Tocoron jail, which serves El Consejo, officially left 16 dead but families say the true figure is higher. Images recorded on mobile phones show bodies, some decapitated and dismembered, being piled onto a pickup truck.

Outsiders are denied access but in a phone interview one Tocoron inmate, Luis Viña, related atrocities from his 16 years at different prisons. Speaking machine-gun fast between gulps of air, exhilarated by contact with the outside world, the convicted rapist and murderer's litany of horrors included beheading a rival and eating his brains: "He was making problems for me." Viña is not insane: it was an effective message to others not to mess with the cannibal. He has been forsaken by his family but not Jesus, he says. "With God's help I have recovered my values."

When Ospino led about three dozen footsoldiers in 2003 into the rehab programme, which is called Alcatraz, the Cementerio gang briefly disappeared. Instead of robbing and getting high, members went camping, took training courses and grappled with rugby – an idea of Vollmer's to make them equal through learning an alien sport. After graduating, many found jobs as security guards, bottlers and cooks on the hacienda. "You could see their faces physically change," says Vollmer, a Venezuelan anglophile of German descent. "At first the muscles were really tense, and gradually they relaxed."

Alcatraz is a success story: local murder rates fell, attracting interest from the government and Harvard academics. Graduates such as Juan Silva, 30, a former drug addict, turned their lives around. Watching him at home playing with his three young children, dog and cats while his wife makes lunch, it is difficult to imagine him killing. But he did. "When I think back to that bus driver, it hurts. I think of him as a father and a husband, a human being. I cut his throat with a bottle. It was almost an obligation in the gang to show you were hard, to earn respect. I was 15, maybe 16."

One day, aged 19, Silva mugged a bus passenger for her jewellery. He fled on foot and caught another bus, conscious of a police patrol ahead. Seated behind him was a pretty girl. "I wanted to say something cute to her but the only thing I could think of was: 'Will you hide my revolver?'" he recalls. The girl smiled and hid it in her purse. Love was born. The couple dated, married and Yainna, now 27, laughs at the memory. "What can I say? I liked him." Evangelical Christians, they pray every day. When not bottling at the rum factory, Silva coaches football and at night studies mechanics.

You hear a similar story from Williams Duran, 30, a wiry, cheerful bundle of energy. "I was a drug addict and thief for so long I robbed everything. But now I'm straight, have a job as a cook, a wife, two kids. I've even got furniture and a bank account."

It would be nice to end the story there but the Cemetery gang, along with other groups that entered rehab, has revived. Shops and houses have barred windows and fresh bulletholes, and kidnappings plague the area. Luis Yuraima, the 20-year-old son of a shopkeeper, has just been rescued after four days in the hands of the Pelucos, who demanded an $83,000 (£52,000) ransom. How was he treated? Without a word he pulls off his T-shirt: purple slashes and puncture wounds cover his back. Johnny Brito, owner of the La Estación cafe, is downbeat about the future. "Some guys get rehabilitated, great, but there's always a new generation behind them," he says.

When Ospino and others entered the hacienda, the vacuum in El Cementerio did not last long. A new leader, José Daniel Nuñez, aka Kibiri, emerged. It was not out of choice but necessity, says his brother, Richard. "Others were coming into our neighbourhood, threatening people. We had to defend ourselves." Kibiri, by common consent, is exceptionally bright. Top of his class, articulate, wily, he is also, depending how you look at it, incredibly lucky or unlucky. Seven years ago, aged 15, he was ambushed and shot 14 times. He survived and hobbled out of hospital, one-eyed, ostensibly reborn. He entered the rehab programme and befriended those who had shot him. Then he killed them. "One at a time," says Richard.

Kibiri was expelled by Alcatraz, caught by police and jailed. Inside, somebody stabbed him 13 times. Again, he survived. "People think he's made a pact with the devil, that he's immortal," says Jimin Perez, the former police officer who now runs the Alcatraz project. Belief in santería, a voodoo-tinged African-Caribbean import, is widespread, especially among gangsters who pray to santos malandros, holy thugs, for success and survival.

Who else, after all, can they turn to? According to the police many of the gangsters' mothers deal drugs. The head of the neighbourhood association allegedly is the main supplier, with a sideline renting revolvers. The state is largely absent save for the police, and their reputation for brutality and corruption rivals that of the gangs. "We learned that the police are more criminal than the criminals," says the ever-blunt Vollmer. "They outsource crimes to the gangs."

Asked about cops, a group of El Cementerio teenagers stop kicking a deflated football to volunteer anecdotes. They sell you 50 bullets for 400 bolivares ($48), says Carlos Noguera, a topless, chubby 15-year-old. He knows about bullets: more than 30 mangled the features of his older brother, a casualty of the popcorn war. If police catch you with drugs or guns they let you go for 100 bolivares, says Juan Carlos Nuñez, also 15. If you're clean they plant stuff so you still have to pay, says his older brother, Richard. Police accidentally shot dead the Nuñez boys' grandmother while chasing a suspect through their home.

Perez, the grizzled former cop, does not deny any of it and has his own stories, including that of two officers who raped a woman in a park. A request to interview the police chief for Aragua state, which includes El Consejo, was declined on the grounds he did not know the Guardian's "political leanings". Even the likes of Inspector Enrique Aray, a dedicated, honest cop who patrols slums in east Caracas, admits ignorance about gangs. "We don't have good information," he says, peering from his Jeep at shadowy figures who flit across his headlights on a rainy, gloomy night. Later, when two shots crack in the distance, he smiles apologetically. "A .38. It's normal here."

With Darwin Ospino confined to the rum hacienda, and Kibiri in jail awaiting trial for murder, the Cementerio gang's fate hangs in the balance. The popcorn war's origins are petty but if the gang does not prevail, or at least keep fighting, it will forfeit clout, and with it drug sales, to rivals. Kibiri's natural successor is Richard. Smart like his brother, and physically similar, the 18-year-old must decide between taking the crown or breaking with family tradition and pursuing a legitimate career. "He is softer, more gentle," says his mother, Yelitza. A tough matriarch, it's not clear she means it as a compliment.

"I've been in shoot-outs but haven't killed anyone," says Richard, seated in a barely furnished living room. On top of a closet, peeking from folded jeans, is a revolver. He acknowledges it, shrugs. "I won't kill. That's not me." He sighs and rubs the bullet scar on his belly. "This life, always afraid, looking around the corner, over your shoulder . . . it's not good."

Richard has a fantasy that one day the father he has not seen in two years will drop by, share an empanada and take him out to a movie. "We'd just sit there, watching it, with a Coke." His thousand-watt smile lights at the thought. He knows it won't happen. His father lives just one hour away but rival gangs would spot him entering and leaving and assume he was delivering bullets or drugs. "They'd kill him."

Against the odds, Richard has stayed in school and is on the verge of graduating. He is thinking, he says, of becoming a mechanic. A crumbling home on a dusty hill in Venezuela, and a young life that must choose between two paths. One filled with danger, good money, prestige and the chance to "defend" the community. The other filled with long hours, a minimum wage and a lesser but still real chance of getting killed just because some kid flicked a possible piece of popcorn. Which would you choose?