Shaun Bailey (pictured left) said Britain’s fear to acknowledge its racist past is partially why the country’s Conservative (Tories) Party has had a hard time attracting Black voters. “I think most of it is historical,” Bailey told The Telegraph. “It’s actually less about race. Do ethnic minority communities feel like they’re part of Britain? The Tory Party [is] certainly linked to the establishment.”

He added that the Party is also struggling with Blacks because it refuses to talk about racism head on. “We need to speak about race,” he said. “Our weakness is we don’t talk about it. If we don’t talk about it, the other side get[s] to tell everyone what you believe about it. We need not to be afraid about it.”

Bailey serves as Cameron’s lone Black advisor. At one point, he was in Cameron’s inner circle, but alleged he was pushed into a part-time role because he was “different” and asked challenging questions about the government’s motives. Though he acknowledges the party’s shying away from race, Bailey says the U.K. is “not as racist as America.”

The Tories are the largest political party in Britain, headed by Cameron. But they have struggled with courting Black and Asian voters in recent years.

According to an Operation Black Votereport, that may come to haunt Cameron in upcoming elections:

“A radical demographic shift means that Britain’s ethnic minority vote may determine the outcome of the 2015 election, according to research.

A study by the cross-party group Operation Black Vote (OBV) found the number of seats where black and Asian voters could decide the outcome had rocketed by 70% compared with the 2010 election.

Unless all parties and candidates engage with and seek to win BME [black and minority ethnic] support, they could be in political difficulty locally and see their general election prospects significantly set back.”

Some are even claiming that the racial difficulties could lead to the country’s first Black PM being elected by 2020.