'If you get kidnapped, I'm not paying," said my father fondly, which was a strange farewell for someone heading to a literary festival, albeit one with a difference. I was to attend an offshoot of the festival currently taking place in Hay-on-Wye in Wales, except that this one was to be held in Cartagena on the coast of Colombia.

This had been the reaction of everybody who heard I was travelling to the country widely reckoned to be the most dangerous in Latin America. So what possessed Hay, a tweedy literary matron more usually associated with wellies than bikinis, to stage a festival in Colombia, I wondered?

"We came because it became apparent that we could not get Gabriel García Márquez to come to us, and because South America - and Colombia in particular - has an amazingly vibrant literary culture," says its director Peter Florence, whose parents started Hay in 1987. "But we also came because only one story is ever told about Colombia. If we can help to change this, all the better." The story concerns cocaine and kidnapping, and Colombia's war against narco-terrorists and anti-government guerillas. So while in recent years the world's largest poetry festival has been held in the city of Medellin, as far as the rest of the world is concerned, its name is still synonymous with the former drug-lord Pablo Escobar.

It is a violent image that the president, Alvaro Uribe, is determined to change. Taking the slogan Colombia es pasión, his ambitious programme aims to rebrand the country as a land of culture and creativity that is ripe for foreign investment and international luxury tourism - preferably before Avianca, the national airline, introduces direct flights from London to the capital, Bogotá, next spring.

Considered more closely, the choice of Cartagena's old town for the Hay Festival is less bizarre than it seems. As the adopted home of "Gabo" García Márquez, the octogenarian Nobel laureate and Latin America's most celebrated living author, it is an enticing prospect for literary types. With the magnitude and madness of the problems at home and the image problem abroad, it is oddly apt for the Colombian government to use literature and fiction as marketing tools.

I certainly found that the setting for Love in the Time of Cholera more than lived up to its reputation for "magical realism". The cobbled streets and vividly painted houses seemed almost too picturesque to be anything but fiction, and reality became unreal. Finding a live toucan sitting on my windowsill seemed quite normal, as did eating a lavish dinner cooked by Escobar's former personal chef and hanging out with Bob Geldof by the rooftop pool of the former Convento de Santa Teresa. No wonder the Spanish conquistadors felt so overwhelmed by magic, witchcraft and blasphemy that they built the imposing Palacio de la Inquisición, now a museum, in the central Plaza de Bolivar.

The inhabitants of this 16th-century Disneyland are as colourful as the bright ropes of bougainvillea that tumble from the carved wooden balconies. On every corner, stout Carmen Mirandas wearing the voluminous traditional costume of nearby Palenque (a town of freed slaves founded in 1603) sell fruit from the bowls on their heads, while smiling señoritas beckon the unwary into shops piled high with gleaming emeralds and Inca gold. Locals and seabirds sun themselves in the the Plaza de los Coches, once the slave market, while children buy sweets from the stalls lining the arcaded walkway known as El Portal de los Dulces.

At night, the narrow streets fill with people and the warm air fills with music and laughter. Men with slicked-back hair, fat cigars and surgically enhanced girlfriends 2ft taller and 30 years younger relax in the buzzing bars and upmarket restaurants such as La Vitrola, a place redolent of Old Havana, with whirring ceiling fans, snake-hipped waiters and oversized wineglasses, all hidden behind an anonymous wooden door.

As a Spanish possession, Cartagena was an appealing prize for British buccaneers such as Francis Drake. In the 16th century alone the city's forts and turrets endured five pirate sieges, none of which managed to breach las murallas, the impressively thick walls that encircle the old town.

This year's Hay line-up included the suitably piratical figure of Christopher Hitchens, known as el contradictor in the South American press. Blame the rum punch, or the magic of Cartagena, but inhibitions melted as fast as the ice in our glasses. Among other indiscretions, he revealed that he and Bill Clinton - the man who memorably called the Hay Festival "The Woodstock of the Mind" - had shared a girlfriend at Oxford who is now a radical feminist lesbian.

Later that evening, in the magnificent Plaza de la Aduana, Geldof played a blinding set for an ecstatic audience of 5,000, which included Francisco Santos, the fiery vice-president of Colombia. A passionate supporter of the rebranding initiative, Santos last year challenged Europe to take some responsibility for the havoc that the drug trade has wrought in his country, saying, "Every gram snorted is a line tainted in blood." In August this year, the Hay Festival is co-hosting Bogotá 39, a celebration of Latin American literature that is taking place in the Colombian capital. The event is part of a programme designed around Bogotá's coronation as World Book Capital by Unesco.

When asked what it was like to be Colombian, García Márquez replied: "The problems that we have are so vast, so deep and so terrible that we are condemned to originality." According to Peter Florence, this originality is being channelled into some of the best writing in Latin America. "These young authors have lived through war and paramilitary intervention. They are the voice of those who have survived, and as creative writers they can often say things that journalists can't," he says.

The Colombian government is also hoping the event will introduce more visitors to the capital city. Anywhere else in the world and Bogotá's picturesque old centre, world-class museums and sights such as the mountain shrine of Monserrate would be attractions enough. These days, unlike five years ago, it is even safe to drive 30 miles out of the city to visit an extraordinary cathedral that was hollowed out of a disused salt mine in the town of Zapaquira.

But it is Cartagena that is Colombia's greatest asset in its rebranding campaign. Its history is as fascinating as it is colourful, its people are among the most welcoming - and beautiful - on the continent and as both a Unesco World Heritage site and Colombia's St Tropez, it has everything that luxury travellers seeking an unspoilt destination could hope for.

At times its upmarket glossiness feels almost claustrophobic. But walk just a couple of blocks away from El Centro and you will find yourself in a living, breathing, working society, where the smiles are genuine and less desperate to please. This is the Colombia that visitors should learn to see.