On Friday Jamaica's most notorious gangster will appear in court in New York to be sentenced for racketeering. By American standards, the 20 years Christopher "Dudus" Coke is facing may not sound terribly long, but his sentence will draw to a close one of the most explosive criminal cases in Jamaican history.

The country was almost brought to its knees two years ago, when Coke's armed supporters flocked to the slums of Kingston to fight security forces battling to arrest and extradite the don to the US. A state of emergency was declared, buildings were burned, police stations bombed and more than 70 lives lost in the bloodbath.

Evidently, it takes a lot to convict a don. In Coke's case it took the full force of American might – and most Jamaican murderers do not face anything like that. It's quite difficult to convey the degree to which crime dominates life for many on an island home to fewer than three million people, of whom roughly 1,500 are murdered each year, making it one of the most murderous nations on the planet. So prolific are the killings that one newspaper's fortnightly crime round-up subdivides them into categories: chopped to death, burned to death, poisoned and so on. There is also a category for all those shot by the police, and by the early 2000s the figure was averaging three a week. The history of corrupt and incompetent policing has been a national tragedy for Jamaicans, but they aren't the only ones. What happens in Kingston will usually find its way on to the streets of Miami, Toronto, New York and London.

Which is why Mark Shields, a detective chief superintendent with the Met, was dispatched to Jamaica back in 2004. Officially, he had been invited by the Jamaican government. "But I suspect it was with one arm up their back," Shields offers with a knowing smile. The UK government was spending a fortune trying to help Jamaica contain its crime problem, but was "extremely concerned about a particular individual".

That individual wasn't a gangster, but a senior Jamaican police officer, Reneto Adams. Adams and his team had distinguished themselves by killing four members of the public in a rural house in 2003. Adams insisted they had died in a shootout, but such was the suspicion surrounding the case that Shields was seconded to investigate. "What I thought would be a two-month assignment turned into two years. And throughout, we were told by everyone, you will never get Adams charged, and if you do he will never ever be convicted."

They were wrong on the first count – but right about the second. "Unfortunately," Shields says, choosing his words with care, "the jury was not persuaded."

Shields will not say so, but everyone I know in Jamaica takes it for granted that the acquittal was essentially corrupt. But Shields had persuaded four officers to testify for the prosecution and on that basis he was invited to stay on in 2005 and become the country's deputy police commissioner, bringing three assistants from UK police forces with him. More than 40 years after Jamaica's independence, the British were back in charge of law and order.

Shields's appointment created quite a sensation on the island. At 1.96 metres (6ft 5in) he would cut a striking figure anywhere, and Kingston was electrified by this glamorous foreigner who published his mobile phone number in a national newspaper, and invited people to call him directly to report crime. If Jamaicans were taken aback, Shields was pretty shocked himself by what he found in his new job.

"I went to homes where two or three generations were just charred remains in a burnt-out house. I've been to more shootings than you can imagine, and more funerals of police officers murdered on duty than I'd ever imagined possible. I remember a police officer's five-year-old daughter taking a bullet in the chest."

If anything, the police service itself was even more shocking. "When I first got here, there was a very inward-looking, nepotistic culture. They were hated by most of the public in Jamaica, because fatal shootings were running at a ridiculously high rate, corruption was out of control, from top to bottom. Anything from allowing drugs to be brought on to the island, and turning a blind eye for a cut, to police officers contracted to kill other criminals, anything you can think of, they did."

Even the officers trying their best were struggling in a system that would have looked old-fashioned a century ago. "It was appalling. An exhibit such as a bullet fragment would be put into a paper brown envelope, and then they would get a red wax seal and stamp it on the back like something out of the Napoleonic war. I'm serious. So you would have this old envelope with a Napoleonic seal on the back, and that's your exhibit." Fingerprints were stored on cards, with no digital database; crimes were laboriously recorded by hand in big old dusty ledgers. "They would just say, that's how we do it." To make matters worse, he adds, "Back then policing was a job you took if you couldn't get any other job. Nobody with brains became a police officer, or not many; it really was a class of person that shouldn't be becoming police officers."

The international operation against Coke was already under way under Shields's watch, involving the US, Canada and the UK. Coke's stronghold was Tivoli Gardens, a notorious district of Kingston where police seldom ever ventured. "I wanted to treat it like any other community, but it was very difficult to do so because the backlash was such. Why was that? Because four years earlier Adams had gone in there and more than 20 people had been killed. Not many communities had lost 26 people in two days, as had happened when Adams and his men went in there. So of course I can understand the sensitivities. But we were forced to treat it with almost kid gloves."

It's very hard, Shields explains, for outsiders to appreciate the sheer power a don like Coke wielded over his community. "It's criminal terrorism. People literally live in fear. If they run a shop, they have to pay protection. If the don wanted their youngest daughter, they would have to give her up so he could take her virginity. The community was completely under the control of the local don, and the police were deeply frightened about going in there."

Following the killing of four officers in 2005, Shields learned that the gunmen were hiding in Tivoli Gardens, and mounted an operation. "Within a matter of minutes, Bruce Golding, the local constituency MP and then leader of the opposition, "was on the streets, along with the mayor, saying what are you doing here, why are you invading our community? They said that to us, the police." Shields shakes his head in disbelief. "Of all the dons, Coke was in a league of his own."

Golding subsequently became prime minister, and his reluctance to extradite Coke to the US led to the street war that broke out when the don was finally taken. Shields was often approached by politicians who seemed strangely keen to befriend him – "and I was never really sure what their motivation was. We had suspicions over particular politicians, who we thought were too close to criminals."

Others in authority weren't wildly enthusiastic about supporting Shields. "Well, without going into too much detail – because I still live here – some police officers definitely tried to undermine our work. Slowing processes down. Not turning up to meetings. Refusing to promote officers we were happy with. Some senior officers literally wouldn't even speak when we entered a room. It was a small vociferous minority who just didn't want us there."

Any worries about how the public would receive him, however, turned out to be baseless.

"From a white liberal academic person from the UK's perspective, the worst thing that you could ever do is send a white police officer into what is a majority black community, to do what I did. So of course I thought about it, and I wasn't so naive to think a white man might not look like a reinvention of colonialism. But in fact it wasn't like that at all. Jamaican perceptions of what we were doing here were completely different to the perceptions we build up in our white, middle-class, educated, liberal England. What they wanted were straight police officers who would speak to the public. And that started a change in the relationship between the JCF [Jamaican Constabulary Force] and the public."

According to Shields, the transformation since he joined the force has been astonishing. "We've got a far more professional police service than we've ever had before, with more public confidence than ever in the past. They have far more resources. They investigate murders properly because they have proper major investigative task forces. The scene of crime officers have been trained to a first-world standard, and the forensic laboratory is far better than it's ever been. We now have more graduates than ever before signing up. They see it's a professional body. A critical mass of people in the organisation now have integrity." Corruption among both police and politicians has reduced dramatically, and in 2006 another notorious don known as Zekes was sentenced to life in prison. "That would never," Shields says with pride, "have happened before."

But the biggest investigation Shields found himself leading was one he could never have expected. With the eyes of the world on Jamaica during the 2007 cricket world cup, Pakistan's manager Bob Woolmer was found dead in his Kingston hotel bathroom. The coroner announced he had been strangled, and pandemonium broke out; the Pakistani team was questioned, rumours involving Far East illegal gambling rings began flying, and every day Shields had to face the cameras to explain why the mystery killer was still at large. Privately, Shields was sure he knew exactly why. There was no killer; the coroner had made a mistake. Woolmer had died of natural causes.

"But I had a duty to act on the coroner's verdict. Besides, if I had said it was an unexplained death, and the pathologist said he was strangled, then who is going to be accused of a cover up? Me." He feared he would be accused of being paid off by illegal betting syndicates in Pakistan? "Absolutely." After three tumultuous months, a new team of pathologists finally confirmed Shields's suspicions; Woolmer had not been murdered. But by then, in the eyes of many, Shields was a laughing stock. "It was the most difficult period of my life," he says quietly.

After four years of service, Shields declined to renew his contract, but remains on the island running his own private security consultancy. He has a young daughter with a Kingston radio presenter, and laughs that he had never imagined ending up with a brand new life and family in Jamaica. "But this place, well, it just gets under your skin." Coke's extradition had left a profound legacy in Shields's new home; Golding – who had been prime minister at the time – was forced to step down from office last year, blamed for the carnage that engulfed the capital in the fight for Coke, and the don's fate has also had a staggering impact on crime rates. The murder rate has dropped by 40%, and not one police officer lost his life in the line of duty last year.

"I think Coke's arrest and extradition put the criminals into some sort of psychological spin. They were running scared. I think it sent shockwaves through the country that if they can do this to him then they can do this to any of us. The change in the relationship between politics and criminals, that's also happened." I would guess that, having had no fear of their own authorities for so long, it's the fear of foreign intervention which is now making corrupt officials think twice.

But the Jamaican government recently announced that it would be recruiting no more British officers, and British cuts have slashed spending on trans-border security. Shields is concerned. "Wherever I worked, anywhere in Kingston, I would suddenly come across a young man who would go: 'Alright, guv?' I'd ask him where he came from and it would be Deptford, or Birmingham, or Bristol. There are those in strategic positions in London who say gun crime and gang crime in Jamaica is not a priority any more. And I think it's shortsighted, because if you take your eye off that particular ball, it's going to roll again and be a massive problem."