Met police chief superintendent Dal Babu said there was a lack of will to change within the police service. Photograph: Martin Godwin

One of the most senior Asian policemen in the country is quitting the service on Monday after being rejected for promotion to chief officer rank.

Chief Superintendent Dal Babu speaks four languages, has been awarded the OBE, holds a master's degree and is responsible for soaring public confidence ratings in the London borough of Harrow he runs, but was considered unsuitable to assume chief officer rank because his media interview skills were not deemed good enough.

Babu, 49, has been a prominent and eloquent spokesman for Muslim police officers with wide public exposure, including being interviewed by John Humphrys on the Today programme.

He was refused a place on the strategic command course for the next generation of chief constables despite repeated public pronouncements by the Association of Chief Police Officers (Acpo) about the need to improve diversity in police leadership, of whom just 2.8% are from black or minority ethnic backgrounds.

Babu, who joined the Metropolitan police in 1983, warns in an interview with the Guardian that the poor representation of ethnic minorities in the police service is going to get worse, particularly in senior roles. He says the responsibility for improving diversity lies squarely with chiefs and reveals there are pockets within the police that are still white male bastions – including specialist CID departments and firearms units.

"The police service is certainly a very different organisation from the one I joined when racist name-calling was commonplace," said Babu. "But my sadness is we have gone from 1% to 5% black and ethnic minority officers in 30 years. We have not managed to replicate the communities we serve. Our major cities are majority ethnic minority and yet the police force remains stubbornly white."We have ended up with lots of theory around police and diversity, and what we need is an ounce of action."

Babu, from Walsall, says he experienced blatant racism in his early career. "People would openly call me names and at times it would come to fisticuffs.

"I look back and I think I must have been incredibly determined. There were several times when I thought of leaving, but I loved the job."

While he no longer experiences open racism, he has had many battles. In 2003 he won a landmark case against the Met police after claiming that he failed to win promotion and faced discrimination because of his faith.

His personal file – which he has seen – contains the comments: "This officer is over-racially sensitive". He leaves on the day the Tory reforms of the service continue with the launch of the College of Policing, which will be responsible for improving standards across the service. The policing minister Damien Green has said the college would be a "key driver" for improving diversity in the police.

But Babu said there was a lack of will to change within the police service. During his time as a middle-ranking officer he has pursued several initiatives which have been seen to help – including increasing the number of Muslim officers in the counter-terrorism unit from eight out of 2,500 to 35 under commissioner Sir Paul Stephenson.

In the four years to 2008 only three black or minority ethnic officers had been promoted to chief inspector in the Met police and many black officers were so disillusioned they were trying to organise a boycott of the whole promotion system in protest.

"I spoke to the head of recruitment and organised for a group of senior officers to do one-to-one mentoring," he said. "We met at my house and they explained to the officers what the process was and what was expected of them, it was intensive mentoring and it was at no cost to the force. It resulted in eight out of the 11 officers involved being successfully promoted."

But Acpo rejected his suggestion that the scheme be adopted for all senior ranks. "I was flabbergasted they weren't prepared to consider it," he said. "It cost nothing and had such a positive impact. Now where we are is in a situation where we have got no one on the strategic command course who is black or ethnic minority this year."

Change could be driven by altering the requirements for police recruits, he said. "We can do this within the law, its perfectly legal. If you started looking at language as a requirement for a job in the police, if you say you want applicants to have experience in volunteering within their own community groups, if you said for the Met, you have got to live in London – all of this would mean you would get more applicants who were black and ethnic minority."

Without radical action, he predicts the representation of black people and ethnic minorities within middle- and senior-ranking posts will get worse."I don't see black and Asian officers in inspector ranks or chief inspector ranks. There is going to be a very significant gap."

Babu was turned down for a place on the strategic command course in 2011 and he chose to retire. "Despite everything that has happened I have been incredibly lucky to be a Metropolitan police officer," he said.

"I don't think it is helpful to be bitter but we can make the police service better than it is. We can make sure that it is more representative; we need to think differently."

• This article was amended on 6 February 2013 to change the word "quits" to "retires" in the headline.