As a relative newcomer to this discussion, I have been attempting to get my hands around the debate on the Foolish Four strategy, its progeny and Mechanical Investing in general. I've been mainly lurking in the background, but have asked questions and made some requests from time to time in an effort to grasp the sum and substance of the debate and information shared at this board. Some of my questions seemed naïve, partly to provoke responses and partly for the more obvious reason.

Although responses have been most satisfactory and responders have been most patient, I needed to do more research and reading on the past discussions/posts on my own. I have reached a point now where I need to summarize my view of the debate/discussion and look for responses to refine that view. I present below a summary version of my observations and views.

I. Preface

I have used statistics as a problem solving tool in my former life, worked with statisticians and developed a fascination for statistics over the years, but I am not a statistician. I am aware that amateur statisticians are prone to missing some of the subtleties of methodology that the working statistician readily comprehends. My summary, as a result, should be viewed with all the biases and deficiencies that my background implies. I have avoided details of exact time periods, probabilities and such to keep the summary brief and focused on general perceptions.

II. Basis of Foolish Four

TMF generated the Foolish Four investing strategy as an extension/modification of the Michael O'Higgins published strategy known as the BTD5. TMF proposed/promoted this strategy based on comparing its favorable total returns produced over that of the DJIA for a stated period of time, referred to as the "Discovery Period". Data from this period and used for the comparison are commonly referred to as "in-sample data".

The initial statistical analysis were made by TMF using simple pairwise comparisons.

III. Statistical Analysis And Data Mining Issues

Data Mining (analysis of historical performance returns to find favorable patterns based on criteria/factors that were discovered through sorting and were not based on prior or ex ante theory) raise the requirements from a comparisionwise Type I error rate in hypothesis testing to an adjusted experimentwise Type I error rate, where all the populations used to reach the final criteria/factors are taken into account.

The adjustments for multi-population comparisons need to account for not only the number of comparisons but any dependencies that exist between them. The procedures used for this adjustment, that I found reported on this Board, were those attributed to Bonferroni, Tukey and Dunnett. References were also made to bootstrap methods and procedures by Scheffe. Criticism of the Bonferroni approach were made to its conservative estimate of Type I errors and the resulting increased risk in making a Type II error, i.e. falsely accepting (actually, not rejecting) the null hypothesis.

Other notable discussion, that I found on statistical comparisons, included suggestions to use the nonparametric Wilcoxon signed-rank test to handle observations that deviate greatly from normal distributions and Sharpe Ratio comparisons to take into account differences in risk.

IV. Data Mining Dangers

Datasnooper has posted numerous examples that clearly illustrate how Data Mining can take patterns from randomly generated data/results, that would be incorrectly calculated to be statistically different from the general population at a high level of confidence, if the nominal Type I error rate from a pairwise comparison was not adjusted for all the comparisons made to find the pattern.

My view of the responses to Datasnooper on Data Mining are that the more serious ones do not dispute the dangers of Data Mining or the general approach for analyzing the in-sample data. The disputes come from determining the amount of Data Mining and thus the adjustments required to determine a proper Type I error rate. That Data Mining does exist in developing Foolish Four and its progeny, has been evidenced by several observations and analyses, including comparing the difference in results in the Discovery Period to those from the Pre-Discovery Period, the discovery of the "January effect" which was not considered as part of ex ante theory (not a strategy criteria) and the fact that the strategies are modifications based on analysis of previous strategies.

As an aside, its my view, that since application of the proper statistics to account for Data Mining hinges on the intentions of the person who developed the strategy, the discussion can appear to take on a personal flavor where none is intended or necessary.

V. My Categorizing of Reactions (Real or Imagined) to Data Mining And Comments

1. One general category of reaction points to the dangers of marketing and using a strategy (mechanical investing screen) without cautioning or being cautioned about the potential intended/unintended/understood/misunderstood use of Data Mining. The stronger version of this reaction puts the responsibility for doing the cautioning and revealing the process of Discovery squarely on the shoulders of the strategy sponsor. It also appears to assume that the strategy was developed using Data Mining unless proof can show otherwise.

Comment: I most agree with this camp.

2. A second category agrees that while Data Mining has occurred, it cannot ever be quantified and thus in-sample data cannot be used to statistically compare for significant differences. The statistical difference in their view must await an out-of-sample comparison. I am not sure, but I suspect this camp could be divided into those who would use both Pre-Discovery and Post-Discovery data and those who would use only Post-Discovery data for a defining comparison. Those using the Post-Discovery data do not judge that sufficient numbers of observations will be available for a meaningful statistical comparison until sometime in 2002.

Comments: I have a difficult time with the assumption that if an exploitable inefficiency existed in the market that it would have existed, in effect, forever or that on the discovery and publicizing of it that it would remain exploitable for any long period of time. I suppose the crucial test for this camp would be the existence or nonexistence of Mechanical Investment screens that continue to operate into the Post-Discovery period for sufficient time to statistically say yea or nay about their success.

3. There is a third major category of posters who, I believe, understand the statistical implications of Data Mining, but feel that the Mechanical Investing screen makes common sense and gives reasonable criteria that could have been derived without Data Mining.

Comments: I suppose this camp would change their mind if presented with sufficient evidence of Data Mining in most screens or of a very high percentage of past Mechanical Investing strategies that failed in Post- Discovery testing.

4. A fourth major category appears to want to forego the statistical analyses and accept the strategy on the faith and goodwill of TMF.

Comments: To them I can only say may TMF bless you and keep you.

VI. My Perception of TMF's View on the Data Mining Debate.

Actually it's ??????. But reading between the lines, I perceive that TMF wants to avoid any legal/moral liabilities for giving out explicit investment advice, of which it constantly reminds the listening/reading audience by way of disclaimer. This timidity carries over into a shyness about taking any definitive stand on the current debate. If my perception is correct, a TMF responsible adult should be able to confirm it without legal or moral risk. If my perception is wrong, I would hope to see something definitive soon from them on the debate. In either case, I am looking forward to seeing a neutral TMF summary of the debate.

Announcements

When Life Gives You Lemons
We all have had hardships and made poor decisions. The important thing is how we respond and grow. Read the story of a Fool who started from nothing, and looks to gain everything.

Work for Fools?
Winner of the Washingtonian great places to work, and Glassdoor #1 Company to Work For 2015! Have access to all of TMF's online and email products for FREE, and be paid for your contributions to TMF! Click the link and start your Fool career.