The trend-setting state of California is voting next month on a ballot initiative to legalize pot, also known as Proposition 19. The measure would legalize recreational use in the state, though federal officials have said they would continue to enforce drug laws in California if the initiative is approved.

"What I think is horrible about all of this, is that we criminalize young people. And we use so many of our excellent resources ... for things that aren't really causing any problems," said Elders. "It's not a toxic substance."

Supporters of California's Prop. 19 say it would raise revenue and cut the cost of enforcement, while opponents point to drug's harmful side-effects.

U.S. Attorney General Eric Holder said in a letter, obtained by CNN Friday, that federal agents would continue to enforce federal marijuana laws and warned Prop. 19, if passed, would be a major stumbling block to federal partnerships between state and local authorities around drug enforcement.

His letter was a response to an August letter from several former directors of the U.S. Drug Enforcement Administration urging the White House to block Prop. 19 if it's approved next month.

Elders stressed the drug is not physically addictive and pointed to the damaging impact of alcohol, which is legal.

"We have the highest number of people in the world being criminalized, many for non-violent crimes related to marijuana," said Elders. "We can use our resources so much better."