11 Answers
11

Larry Harris has a chapter on performance evaluation in Trading and Exchanges. He states that over a long period of time, a skilled asset manager will consistently have excess returns whereas a lucky one will be expected to have random and unpredictable returns. Thus, we start with the portfolio's market-adjusted return standard deviation:

Now we can simply determine the probability that the manager's excess returns were luck by plugging this t-statistic into the t-distribution's PDF with degrees-of-freedom $n - 1$. The lower the probability, the more we can believe the manager's excess returns were from skill.

I feel compelled to point out that plugging your t statistic into the t distribution does not give you the probability that the excess returns were from luck. Instead, it gives you the probability that the fund manager would have earned those returns if he was unskilled (your null hypothesis), and if your distributional assumptions are correct (which they're almost certainly not).
–
Chris TaylorSep 19 '12 at 16:30

2

@ChrisTaylor When we say that a fund manager is lucky, we mean that he achieved returns without skill; I stand by my original wording. As for whether the t distribution is a valid assumption, it's fine for this question.
–
chrisaycockSep 20 '12 at 3:09

I saw another interesting approach where some researchers (I lost the names) looked for serial correlation in returns within a group. Interestingly they found more serial correlation in negative results than positive results.
–
BlueTrinNov 1 '12 at 14:20

I've purged the comment thread between Freddy and Phil H. This site isn't a discussion forum.
–
chrisaycockFeb 20 '13 at 18:52

+1. Statistical comparisons always require some baseline. As in The Lady Tasting Tea, it's up to you to establish a sensible baseline. One might be able to improve upon completely random baselines by comparing only amongst similar subsets, for example. Or maybe it's better to keep hands off so one doesn't have to justify this or that particular weighting scheme. But then what justifies the uniform weighting scheme? Etc.
–
isomorphismesMar 15 '13 at 22:36

+1 the random portfolio approach offers a good test of the skill of a fund manager. In addition, it is possible to augment the test with the constraints that a manager might be facing, such as asset weights or turnover.
–
FelixSep 29 '14 at 14:27

Below is some code that I used recently to illustrate luck (and con-games). The story went like this:

I'll dream up your lucky lottery number for 2010.....let's say it's 20639. The number doesn't matter because we're going to use that for the seed of a random number generator. Then, I'll take the first three digits of your lucky lottery number (206) and reverse them (-.602) and use that as a multiplier on that random number. Since the S&P500 started off in 2010 at about 1100, I'll start the model at that level. Here is the code:

This is all meaningless BS, but it took a while for some people to untangle luck from the con. The really hard part is to convince yourself that your "skill" actually found something real. Something "real" is a very rare event in the world of investing.

Read Fooled by Randomness by Nassim Taleb. In a nutshell, he says that you can only tell the difference by understanding the risks that were taken. Lucky investors can win for many years before blowing up. Even if he doesn't blow up, there is no way to know what might have happened if the risks turned out badly.

I remember an article from graduate school that describes a methodology for measuring the true timing ability of a money manager. I don't remember the name of the article nor the name of the author, however, I do remember some of the details of the article. Maybe someone else has run across it and would be kind enough to post the appropriate reference.

Let's assume that a manager has the ability to be either in cash, earning the risk-free rate, or a long position in a basket of stocks. If the money manager had superior timing ability, he would be in the basket when the basket was returning greater than the risk-free rate, and he would be in a cash position when the basket is returning less than the risk-free rate. What you basically have is a return profile that looks a lot like the payoff of call option.

If you plot market return on the x-axis and manager return on the y-axis, the return should be flat at the risk-free rate for everything to the left of the risk-free rate on the x-axis/ At the risk free-rate on the x-axis, the return should be a 45-degree line up and to the right of the diagram. Over time, you measure manager return against market return, and if he is any good, you should see the call option payoff diagram being roughly drawn out.

Martijn Cremers and Antti Petajisto have a series of papers using the concept of "Active Share," a new measure of active portfolio management which represents the share of portfolio holdings that differ from the benchmark index holdings, to evaluate mutual fund managers. They find that the most active stock pickers have outperformed their benchmark indices even after fees and transaction costs. In contrast, closet indexers or funds focusing on factor bets have lost to their benchmarks after fees.

Bottom line: when separating skill from luck, concentrate on those managers that actually try to differentiate themselves from the crowd. Also, the more fine grained your strategy is, the more likely it is to represent skill over luck.

Experience tells me that the better ways to get a feel for whether their strategy is based on something more than luck are amongst:

1) `getting to know your traders' -- have a chat, pick their brains, try to get some insight into their methods;

2) see how hard the market has been -- check whether you have just been part of a bull market which basically made pretty-much every strategy a winner, and discount performance accordingly.

I would be skeptical of anything too mathematical for measuring outperformance, since even a badly-designed strategy may have performed well.

I see an analogy with driving: many drivers will successfully get from A to B, but you'll get a much clearer picture of the sorts of risks a driver takes by sitting in the car with them, rather than trying some mathematical analysis of the race. Make sense?

Thanks for your $0.02. Problem with what you're saying is: #1 you can only sit down and chat with so many managers, plus only so many will even be willing to sit down and chat with you. For #2, that doesn't sound like a very sound systematic long-term metric. The point of the question (and the whole site, for that matter) is to get beyond vague generalizations and intuition and use some math to solve these problems. Your answer basically amounts to saying we should give up on trying to answer this question using quant techniques.
–
Tal FishmanJan 30 '12 at 21:05

1

@TalFishman Both of your points are fair, but your summary of my answer is not. I do not propose that we `give up'. Au contraire, I would encourage the quant community to get out there and talk to their traders; we quants generally don't know enough about what traders do (there are questions in quant.stackexchange that express this frustration). Once we get a better understanding, let's put it down in mathematics.
–
RobertJan 31 '12 at 8:44