"I think this is the beginning of the conversation on legalization, not the end," former Obama Drug Policy Advisor Dr. Kevin Sabet told ABCNews.com today.

"When you have the governors of both states [opposing it] as well as the president and Congress, who has already determined that marijuana is illegal, this is not going to be a walk in the park for marijuana enthusiasts," Sabet said.

In a groundbreaking move, Colorado and Washington voters passed referendums legalizing marijuana for recreational use. The drug is still banned under federal law.

Colorado's Proposition 64 to the state's constitution makes it legal for anyone over the age of 21 to possess marijuana and for businesses to sell it.

"The voters have spoken and we have to respect their will," Colorado Gov. John Hickenlooper said in a statement. "This will be a complicated process, but we intend to follow through. That said, federal law still says marijuana is an illegal drug so don't break out the Cheetos or gold fish too quickly."

A similar measure on the ballot in Washington State legalizes small amounts of marijuana for people over 21.

Even though the measures have passed, they are likely to meet legal challenges very quickly.

Sabet predicted that the federal government would find a way to make clear that the passage of the amendments violates federal law and they won't "take this lying down."

"I wouldn't advise anyone to toke up just yet. This is going to be caught up in the courts. This is a to-be-determined situation," Sabet said. "The government has multiple avenues. They can wait until it's implemented, take action before it's implemented, reiterate what federal law is, send warning letters."

The avenue the federal government will choose remains to be seen, Sabet said, especially since the passage is unprecedented.

"If you look back, the only precedent is the medical marijuana situation," he said.

In 2005, the Supreme Court by an 8-0 margin struck down a California law that legalized medical marijuana in the state. The Court said Congress had the power to criminalize marijuana under the Commerce Clause.

He said that it may take a state's move toward implementation to spur the federal government into intervening.

"I think you're going to see very soon a response from the administration," he said. "It's completely premature for any legalization advocate to be celebrating at this point."

Nevertheless, the amendment's strongest proponents in Colorado celebrated the win.

"The people of Colorado have rejected the failed policy of marijuana prohibition," said Brian Vicente, co-director of the Yes on 64 campaign, according to ABC News' Denver affiliate KMGH. "Thanks to their votes, we will now reap the benefits of regulation."

"It would certainly be a travesty if the Obama administration used its power to impose marijuana prohibition upon a state whose people have declared, through the democratic process, that they want it to end," Vicente said.

"The Drug Enforcement Administration's enforcement of the Controlled Substances Act remains unchanged," the DEA said in a statement today. "In enacting the Controlled Substances Act, Congress determined that marijuana is a Schedule I controlled substance. The Department of Justice is reviewing the ballot initiatives and we have no additional comment at this time."

A similar ballot issue to legalize marijuana in Oregon did not pass. In Massachusetts, voters approved legislation to allow marijuana for medicinal reasons, joining 17 other states that allow it.

When asked if the federal government may try to quickly quash the amendments as a way to prevent a potential future domino effect of other states following in the footsteps of Colorado and Washington, Sabet said a failed legalization could actually set the movement back.

"A lot is going to ride on what happens next in these two states." he said. "This very well may backfire because if this does not turn out so well, if implementation does not happen, the donors and millionaires that donated for this to happen may pause when doing it in other states."

"I think people should just pause before celebrating this," Sabet said. "The story is just beginning."