Where is the Stanford Ponzi loot? This is the man who knows

Victims of the massive Allen Stanford Ponzi scheme got a rare bit of good news Thursday. Some $18 million in Stanford funds that had been parked in Canada was returned to the U.S. for distribution to investors.

But the payment is tiny in comparison to the $5 billion in actual losses from the scam, and the court-appointed receiver who has spent the past five years searching for Stanford's billions tells CNBC it is unlikely investors will ever recover much because for the most part, "the money is gone."

"I think about the victims every day," Dallas attorney Ralph Janvey told CNBC in an exclusive interview, his first since being appointed five years ago.

"They believed what they were told. And I will say publicly I don't think the victims should be blamed for anything," Janvey said. "They made an investment based on what they were told. What they were told was wrong and it was fraud."

A federal judge placed Stanford's global financial empire in receivership—and under Janvey's control—after the Securities and Exchange Commission filed suit in 2009. The suit accused Stanford of running "a fraud of shocking magnitude" based on bogus certificates of deposit.

Stanford's brokers sold the CDs—issued by his offshore bank in Antigua and Barbuda—to investors around the world. But a federal jury found in 2012 that most of the funds went to support Stanford's lavish lifestyle. Stanford is serving a 110-year prison sentence, and appealing his conviction on 13 criminal counts.

Comparisons to Madoff Ponzi scheme

Since taking over as receiver, Janvey has been searching the world for assets as well as enduring frequent comparisons to the much more successful recovery effort in the Bernard Madoff Ponzi scheme.

"I don't minimize the Madoff fraud and I don't want to minimize the Madoff tragedy," Janvey said. "But Stanford was much more complicated."

That is reflected in the fact that while Janvey's counterpart in the Madoff case—trustee Irving Picard—has said investors could eventually recoup all of their principal. Janvey says the best case scenario for Stanford investors is "pennies on the dollar."

So far, Stanford's 28,000 victims—more than 10 times the number of victims in the Madoff case—have received next to nothing. Janvey received court approval last year to begin making his first distribution to clients. The $55 million being paid out amounts to less than one penny on the dollar.

To date, Janvey says, he has recovered only about $263 million. But nearly half of that—around $120 million—has been eaten up by expenses including $57 million just to wind down Stanford's global operations including 130 companies and 3,000 employees in 30 countries. Janvey says the money, including fees for him and a team of experts, "had to be spent" because of the complexity of the fraud.

"Stanford was spending $33 million a month on expenses," Janvey said. "We had to shut that down. It's also going to litigation; with the goal being what we spend is to bring money back into receivership to distribute to the victims."

Janvey has filed dozens of lawsuits seeking more than $680 million from banks, law firms, politicians and charities that benefited from Stanford's largess. For the most part, they have been unwilling to give any of the money back.

"We had to sue the Democratic and Republican committees to get back over a million dollars.That shouldn't happen," Janvey said.

It took a federal appeals court ruling last year to get the campaign contributions returned, after the committees challenged Janvey's standing to claw back the money.

"Stanford had no right to give the money for political contributions, and they had no right to give it away," Janvey said. "We have other politicians who still have money who haven't given it to us yet."

They include President Barack Obama's 2008 presidential campaign, which received $4,600, according to court filings, and Senate Minority Leader Mitch McConnell, R-Ky., who received $2,500. Janvey said the amounts are too small to justify the cost of litigation.

"Another example is the University of Miami," Janvey said. "They got $6 million to do a study of fish and coral in Antigua. That did not help the victims. That money should come back to the receivership estate so it can be distributed to the victims."

Janvey is also a party in suits against five banks that did business with Stanford, alleging they knew or should have known about the fraud.

"They were on notice and they did nothing about it. And they enabled that fraud to go on for years and years and years," he said.

The cases are similar to allegations against Madoff's primary banker—JPMorgan Chase—which earlier this year agreed to pay more than $2 billion to head off criminal charges. But Stanford's banks, HSBC, Societe Generale, Toronto Dominion, Trustmark and Bank of Houston, have all moved to dismiss the allegations. Janvey said the Justice Department should be investigating Stanford's banks the way it did with Madoff's.

"I would encourage the government to look at the banks," Janvey said. "We are here to assist them if they want our assistance."

Further frustrating Janvey's efforts is the fact that the Securities Investor Protection Corporation (SIPC) has refused to provide coverage to Stanford victims—another contrast to the Madoff case, where the SIPC has not only paid victims but also covered the trustee's expenses.

Money hidden offshore?

Stanford's deep political connections as well as rumors—mostly cultivated by him—that he worked as a government informant have led to speculation he may have some money hidden offshore. But Janvey says that is not the case.

"And we've looked. It's one of the things we did in the early days of the receivership. It doesn't exist," he said. "We've traced the money. The money is gone. And that's very frustrating for the victims, I understand that. But the money was gone before we came in."

Janvey says the money that was not redeemed by Stanford investors "went to his spending."

"He had to buy the reputation that he—quote—'earned,' and he used victims' money to get there."

Some $208 million is frozen in Swiss banks, with about two-thirds of the money to go into the receivership controlled by Janvey, and the rest to go to court-appointed liquidators in Antigua. But much of that money is contingent on Stanford's criminal appeal. Stanford, who is representing himself, recently filed a motion from prison asking for another six months to file his initial brief.

Janvey estimates it will take another five years of litigation before he has exhausted his recovery efforts.