Hedge fund journals everywhere—this one included—routinely rely on and trumpet hedge fund indices as accurate barometers of how the industry is performing. But a new study has revived concerns that indices are anything but an unvarnished look at hedge fund returns.

According to the Edhec-Risk Institute, hedge fund indices routinely overstate industry performance. The study reads, "the performance of multi-strategy indices whose portfolios included illiquid strategies was extraordinarily overstated after mid-2008."

"It is more and more difficult to justify the use of non-investable composite indices as benchmarks unless we can suggest a practical and easy-to-implement solution that could substantially reduce the biases that overstate their performance, especially in periods of market stress," Edhec head of applied research Felix Goltz told the Financial Times.

Just how much do indices overstate performance? By a lot, according to some players. Yale University Professor Roger Ibbotson and Peng Chen of Ibbotsen Associates found that the compound annual return of the Lipper Tass database was just 8.98%, compared to a reported 16.45%. Michael Azlen of Frontier Capital said his firm cuts single-manager index performance by 4% a year and fund of funds indices by 1%.

Indices are plagued by three types of biases, according to critics. End-of-life reporting bias occurs when a hedge fund in a death spiral simply stops reporting its growing losses to indices. Speaking of death spirals, indices are also accused of a survivorship bias, with indices showing the returns only of surviving—and presumably thriving—hedge funds and excluding the hundreds of hedge funds that fail every year, many of them due to poor performance. Finally, there is backfill bias, caused when an index allows a hedge fund to retroactively add its historic data after it begins reporting results.

A five-year-old study of the Lipper database, by Princeton University economist Burton Malkiel and Analysis Group's Atanu Saha, show that backfilled returns were an average of 500 basis points higher per year than returns reported live between 1994 and 2003. The same report showed that surviving hedge funds enjoyed returns that were 740 basis points higher than funds that did not survive. And Malkiel thinks things have gotten worse.

"There seems to me to be circumstantial evidence that the survivorship bias during the credit crisis was even larger," he told the FT. "There are still many biases in the data and published reports overstate returns, so what you see is not necessarily what you get."

One prominent index provider, Hedge Fund Research, defended the benchmarks. President Kenneth Heinz told the FT that his firm works hard "to mitigate the effects of those biases." Those efforts mean that HFR's indices avoid backfill and survivorship biases to a great extent.