The maverick Johnson remains barely known to most ordinary Americans but is expected to take votes from Obama – and Mitt Romney – in Colorado on the back of enthusiasm for a separate vote on November 6 over whether to legalise marijuana.

Hundreds packed a university auditorium to hear him speak on Monday night. Dozens who did not fit inside lined the corridors, ears straining to catch the words. All cheered and whooped Johnson's calls for social tolerance and a radically downsized government.

"There's a lot of support for my message here. You can feel it," he told the Guardian afterwards. "Young people who are disappointed with the president are listening to what I have to say."

Some Democrats, haunted by Ralph Nader's torpedoing of Al Gore in 2000, fear Johnson may damage Obama more than Romney and hand the state, and the White House, to the Republicans.

Polls have locked the president and Romney in a statistical tie, with one this week giving each candidate 47% each. A few thousand or even just a few hundred votes could swing the state.

"I want you to make sure you sign, seal and deliver this election to Barack Obama," Bill Clinton told a rally in Denver on Tuesday night. The outcome could pivot on a handful of votes in Colorado, he said.

No one doubts Obama will sweep Boulder and Colorado's other liberal strongholds but the question is: will the margin be enough to offset Romney's advantage in conservative parts of Colorado Springs and the Denver suburbs?

Johnson has declared a target of 5%, saying this would wake up Washington to the libertarian message, but in such a tight race analysts say even 0.5% could prove decisive.

The lack of detailed polling on Johnson has made Republicans and Democrats equally nervous because he could plausibly sink Romney or Obama.

"The few surveys that have captured his level of support show that he draws from both candidates – the young people from Obama and he draws the Libertarians from Romney and those who may have been for Ron Paul," Floyd Ciruli, a Denver-based pollster, told DailyCamera.com. "But none of the samples have been big enough to show which candidate loses more votes."

Two Denver Post polls showed Johnson drawing less than 3% and not changing the margins between Obama and Romney. A poll from Public Policy Polling gave Johnson 2% and said he drew slightly more support from Obama than Romney, but not enough to make a difference.

For Obama to invest precious, dwindling time in Boulder – his third visit of the campaign – underlined concern about Johnson.

"He is trying hard to peel student votes away from President Obama around the issue of deregulation of marijuana. With Colorado being so vital and so close at this point, we can't let that happen."

Democrats handed out flyers near the auditorium and, judging from the pro-Obama slant of several questioners, joined the audience, but there was no dampening Johnson's spark.

A youthful 60, wearing a black jacket over a CND T-shirt, Johnson bounded onto the platform and started with a brief resumé of his life: entrepreneur, elected governor of New Mexico as a Republican in 1994, re-elected in 1998, vetoed apparently more bills than the country's 49 other governors combined, competed in Iron Man triathlons, climbed Mount Everest. And now running for the White House with slogans such as "I'm free" and "The people's president."

Government, he said, should keep costs low and stay out of the bedroom. "You know you're a libertarian when you hate speed limits," he said, earning cheers.

He promised to end the drug war by legalising marijuana as a first step to legalising other drugs. Colorado's amendment 64, which would allow anyone over 21 to use marijuana, could set an example for America and the world, he said, louder cheers.

He advocated non-interventionism overseas. "I'm the only candidate that doesn't want to bomb Iran. And that wants to pull our troops out of Afghanistan tomorrow."

Johnson condemned the Patriot Act and the National Defense Authorization act as civil rights violations and called Obama a feeble advocate of gay rights and other liberal causes. "I agree with everything he says. He was the great hope. But none of it has transpired."

Johnson also said he would abolish the federal reserve, the department of education, the IRS, income tax and corporate tax. A smaller government and a federal consumption tax would tame the deficit.

"Some say a vote for me is wasting your vote. Wasting your vote is voting for someone you don't believe in. So I say waste your vote, vote for Gary Johnson. If everyone does that, I'll be the next president of the United States." A standing ovation followed.

Questioners who challenged his hostility to Medicare and student loans – supposedly a Democrat trump card here – gained little support from the audience.

Johnson, who is on the ballot in 48 states, told the Guardian he had smoked pot but gave it up, along with alcohol, when he became an athlete. He said he appeared to be drawing more support from Obama than Romney in Colorado, New Mexico and Nevada, and vice versa in Michigan and Virginia.

Jack Dille, 21, a humanities major, endorsed Johnson. "Romney and Obama are like milquetoast, just boring. And think of the drug war, it's ridiculous. People say I'm wasting my vote, but if I can't be idealistic now, when can I be?"

Dille is lost to Obama, but the president's return to Boulder may sway undecideds like Kevin Blair, 28, an international affairs student who disapproves of both Romney and Obama. "Our drone strikes are killing civilians, it's appalling. I'm leaning towards Obama but it sucks that we're stuck with these two schmucks."