HOUSTON (CNNMoney.com) -- Despite widespread public perception that speculative investing is to blame for high oil prices, big investors distanced themselves from it Tuesday, saying the recent run up has more to do with strong demand, tight supply, and a desire to diversify instead of trading momentum.

"It's buying into an asset class that has very little correlation to [company stock prices]," said Andrew Safran, Citigroup's vice chairman for global investment banking, speaking at the Cambridge Energy Research Associates' (CERA) annual energy conference in Houston Tuesday. "They aren't really speculators, they are making alternative investments."

With oil prices near $100 a barrel and gasoline hovering around $3 a gallon, speculative investment from banks and hedge funds has taken heat for adding as much as $30 or $40 to the cost of a barrel of oil. The finger-pointing became especially acute in 2007, when crude prices went from around $50 to nearly $100 in 12 months.

"It's pure speculation," longtime Oppenheimer oil analyst Fadel Gheit said in a recent CNNMoney.com story. Nothing has changed from a year ago, he said, he said "Not a single thing."

Consumer rights groups have also lambasted banks, hedge funds and other oil investors, saying the sheer increase in the number of contracts traded on futures markets is resulting in an artificial price premium.

The price run up and increase in trading activity has even attracted the attention of Congress, which recently held hearings on the issue.

"The commodities futures markets have become an orgy of speculation, a carnival of greed," said Sen. Carl Levin, D-Mich., at the hearing last December. "I see no justification for oil to be at $100 a barrel."

But other industry analysts say the premium is minimal, adding only $10 or $15 and providing much needed liquidity to the market.

Citigroup's Safran said there's probably a greater chance oil prices will go down rather than up in the near term, but he still sees interest remaining high.

"This is not a flash in the pan," he said. "Commodities as an asset class are generally attractive to investors. It's here to stay."

Gordon Goodman, a trading control officer at Occidental Energy Marketing, said there's undoubtedly more investment money in oil markets now compared to a few years ago, and he expects tighter regulation on that money in the near future.

"Are hedge funds driving prices in a way that makes them more volatile? There's probably some truth to that," he said. Still, volatility implies prices moving both up and down, and he didn't blame the funds for pushing prices to their recent highs.

Clay Seigle, a director at CERA's Institutional Investor Service, said supplies have gotten tighter over the last year, contributing to what he called a "headline driven" market - basically an environment where there's such a small amount of spare oil in the world to cover a supply disruption that traders bid up oil prices at the first hint of instability in an oil producing region.

"It's backed to a large extent by the fundamental picture," he said "People think prices are going higher."

And for those who think it's only rich Wall Street types getting in on the commodities boom, Citibank's Safran said investors of all stripes have gotten in on the action.

"How many of you have a pension or have given to a university endowment," he asked the room, only half-jokingly. "So you're all speculators."