Dr. Ben Carson told Fox News’ Greta Van Susteren the legalization of recreational marijuana in Colorado should be concerning for Americans, as it is a gateway drug to more dangerous substances.

Carson said on “On the Record” Thursday though there are some benefits to medical marijuana, he worries about the effect legal pot sales would have on American society as a whole.

“It tends to be a starter drug for people who move onto heavier duty drugs – sometimes legal, sometimes illegal – I don’t think this is something that we really want for our society,” he said. “You know, we’re gradually just removing all the barriers to hedonistic activity and you know, it’s just, we’re changing so rapidly to a different type of society and nobody is getting a chance to discuss it.”

Carson said Americans need to sit down and talk about the negative effects of legalizing the drug, but the discussion is considered taboo and politically incorrect.

“We need to talk about how we use it in such a way that it doesn’t result in addition of many, many people in our society,” he said. “Why can’t we talk about these things? That’s what I want to know.”