Fooled by Random Backtesting

Backtesting trading systems on historical data is again becoming popular almost 30 years after it started being used by individual traders due to recent advances in web technology and server speed that allows its online implementation. If one knows what backtesting is all about and its hidden assumptions and pitfalls then its use can be justified. But those who think they will find a trading system for 1-minute bars using backtesting will probably be fooled by randomness and they will only lose their money.

Below are a few excerpts from my (out-of-print) book Profitability and Systematic Trading that some in the backtesting cult did not like at all.

Back testing is a method of analyzing the performance of trading systems, using historical data as input. Such practice became popular because of the argument that “a trading system that performs well during back testing is a better choice over one that does not.” Yet, I believe that this argument has at least one hidden assumption and an empirical shortcoming. The assumption is that history repeats itself and the reaction of the participants to the same information flow is the same. The empirical shortcoming is that the trading system is not a market participant during back testing. One way of dealing with these fundamental issues is to establish criteria that minimize the variation between historical and actual trading results. It is important to understand that the variation between actual and back-testing results cannot be brought down to zero. But since there is no other way of analyzing the performance of a trading system in advance other than by back testing it on historical data, this appears to be a natural compromise.

The empirical shortcoming of backtesting, something that is very serious and the industry that has been set around it avoids to discuss at all costs, is that when a system is backtested it is not a market participant and its positions do not cause reactions. As shown in Figure 1 below, the model of a trading system during actual trading becomes a market participant. This shortcoming can get very serious as the time frame decreases and can shatter the hopes and dreams of those who think they can develop trading systems of any frequency (low or high) for intraday trading via backtesting.

To legitimize backtesting, an additional assumption must be made to overcome the empirical shortcoming, as discussed in the excerpt from the same book, below:

A naïve way of legitimizing the use of back testing is by making the additional assumption that had the trading system been used in actual trading, it would not have affected price direction. In the context of the market structure defined in Chapter 1, the implication of this assumption is that the actions of a trading system in the market do not provoke a reaction by other participants in such a way as to decrease expected profitability. The validity of such an assumption turns out to depend also on the trading time frame considered. For example, in intra­day trading time frames, signals generated by a trading system are likely to provoke immediate reaction from other participants who have the ability to affect intraday price direction. However, as the trading time frame increases, the ability of market participants to affect price direction diminishes fast. The reason is that the influence of fundamental factors on price behavior prevails in medium-to-longer-term timeframes. Thus, some trading systems may exhibit the least variation between historical and actual performance in longer-term trading and the highest variation in intraday trading. But arguments claiming that back-testing results are realistic because trading systems can be assumed not to affect prices are not sound in general.

In other words, what I am saying above is that market makers and market movers will not let the small fish profit no matter what the backtest results were.

And finally:

In practice, then, the variation between back-tested and actual performance can be decreased if the system is used in liquid markets. If liquidity is low, some market participants may be in a better position to move prices in order to pocket the losses of other participants. This is easier to achieve in intraday time frames than in longer-term. Thus, systematic trading can be more effective in liquid markets because it relies on a model that must maintain a minimum performance and that can be achieved only if random effects are kept minimum.

Knowing how to properly develop and apply a systematic trading methodology in speculative zero-sum games and understanding the limitations is fundamental for success. It is important to keep in mind that back testing provides no guarantee of future success but only a gross estimate of the hypothetical historical performance of a trading system.

Now if you have read the above and you still want to keep on trying to develop that 1-minute intraday trading system for ES futures or SPY using a few months’ worth of intraday data, then you will be probably fooled by randomness and backtesting assumptions that are grossly violated in reality. I am not saying it cannot be done in general but it takes much more than an average skill to achieve that.

For more articles about backtesting and technical analysis in general, you may visit our Trader Education section.

If you found this article interesting, I invite you follow this blog via any of these methods: RSS or Email, or follow us onTwitter

If you have any questions or comments, happy to connect on Twitter: @mikeharrisNY