When the previous week I was reading the publication of @KlondikeFX about mt4 vs. mt5, I realized that I had not written anything that I would have found interesting either. Well, let´s try to change this.

In this post, I will introduce the Walk-forward analysis and give an example of a python implementation. The example uses a simple technique of machine learning to be fully operational, but any machine learning model could benefit from this python script.

What is a Walk-Forward.

The best way to learn about Walk-Forward is to read Robert Pardo´s book: “The Evaluation and Optimization of Trading Strategies”

A newly expanded and updated edition of the trading classic, Design, Testing, and Optimization of Trading Systems Trading systems expert Robert Pardo is back, and in The Evaluation and Optimization of ...

In summary, the concept of walk-forward testing is similar to backtest a strategy using ‘in-sample’ data and check the results on an ‘out-of-sample’ period. The difference is that instead of using a single large backtesting period, we split the whole data into smaller chunks to perform consecutive analyses.

For example, instead of optimizing ten years of data and using the eleventh year for testing, the optimization is done first across the first three years, and the system is tested in the fourth. Once this test is completed, the four years window is moved forward one year, and the procedure is repeated until the last year is reached. Figure one graphically shows a non-anchored Walk-forward:

The term ‘non-anchored’ is relative to the fact that the Walk-forward analysis is performed with similar sample size periods. Figure 2 shows the scheme of an ‘anchored’ Walk-forward, where the starting point of the ‘in sample’ set is always the same, and the ‘in sample’ set is increased by the ‘out of sample’ size on each step.

As Robert Pardo states, one of the primary benefits of the Walk-Forward Analysis is to determine the robustness of the trading strategy.

The central benefit of the Walk-Forward Analysis is to determine the degree of confidence with which the trader may anticipate that the strategy will perform in real-time trading as it has in its historical testing.

Another important advantage of Walk-Forward Analysis is to produce peak trading performance as markets, trends, and volatility change. Since the Walk-Forward Analysis provides a method of periodic re-optimization with current market conditions, this often means that it can produce trading performance superior to that of traditional optimization.

In summary, the main benefits of Walk-Forward Analysis are:

Evaluation of the likelihood of a trading strategy performing well in real-time trading

Measurement of the robustness of the trading strategy

Achievement of peak trading performance at a level superior to that of traditional optimization

Maintenance of superior trading performance through more effective adaptation to changing market conditions.

Python implementation.

In this example, our goal will be to predict tomorrow´s Close using today's information: today's Open, High, Low and Close.

1. Import libraries

The first part of the code is committed to import the libraries which we are going to use:

2. Data collecting and preparation for Machine Learning

Collect the data:We are going to get data from the Quandl API. Getting data from Quandl is free, but you should sign-in to get your API-key. Once you have your API-key you should insert it on the following line:

quandl.ApiConfig.api_key = 'Z9xLasdf………xR4d'

Prepare data for machine learning:Supervised machine learning needs labeled features. In other words, our model is going to predict next day Close (Labels) using today’s information (features), and we have to provide the information to the model in a comprehensible way. What we should do is to add another column and insert in it the shifted Close prices we want to predict. The process is illustrated in the following image:

Note that if instead of using the raw data, you want to use the use the difference from step ‘t’ to step ‘t-1’, you should insert the following lines before ‘return data’ :

data = data.diff()
data = data.tail(len(data)-1)

3. Choose Walk-forward parameters

Once we have imported the libraries and have prepared the Quandl data, we must choose the parameters of our Walk-forward analysis. The parameters to choose are the following:

Ticker: the ticker of the asset we want to analyze. In our toy example we are using Goldman Sachs. Ticker=’GS’

n_samples: the number of instances we want to use for optimizing plus testing the strategy. Note that for a non-anchored walk-forward this parameter states the number of instances of the first run.

test_percentage: percentage of instances from the n_samples that we are going to use to test the strategy. For example, if we set n_samples to 2000 instances and test_percentage to 15, we will be optimizing the strategy on the first 1.700 instances and testing on the last 300.

anchored: Boolean. Whether we want to use or not an anchored Walk-forward. Default is set to False.

Ticker = 'GS'
n_samples = 2000
test_percentage = 10
anchored = False

4. Walk Forward implementation:

The next function implements the walk-forward with the parameters we have already choosen:

5. Building our machine learning model:

The following code implements a simple linear regression model on each ‘in-sample’ data (train data) the walk-forward function yields, and returns the predictions of the ‘out-of-sample’ data (test data). Note that I am using a linear regression model as a toy example, but here is where you should implement your own model, modelling the ‘in-sample’ data to yield the ‘out-of-sample’ predictions.

Run the script:

Finally, we run the script calling all the previous functions. The DataFrame ‘alldf’ contains the real out-of-sample data along with their predictions.

alldf = wf(Ticker,n_samples,test_percentage, anchored)
print(alldf)

Notes and caveats:

This post is aimed just to explain the way a Walk-forward analysis works.

This code is not bullet-proof. Please revise and understand it carefully if you want to use it for your own analysis.

The last 'out of sample' size could be larger than the previous ones to reach the end of the dataset.

Data should be normalized to improve performance.

The code used the simplest machine learning model at hand to explain the inner workings of a Walk-forward, but any model could be used (Random Forest, SVM, neural nets, etc.). For example, if you want to use a Random Forest model, you just need to import the model from the scikit-learn library:

Wow!! The predictions are so close to the real data that we can´t even distinguish the real data from the predictions, magic!Well... not really… if we take a closer look for example to the last 8 instances we will see the following:

As we are getting yesterday`s features to predict today’s close, the predictions are just kind of ‘following’ the real price. There is no magic here and the strategy won´t yield positive returns. Here is where you should work with the machine learning models to get better predictions.

The whole code will be post soon on my webpage: www.tradingmove.com (yes, I know I must redesign it, just need to find time…)

What I especially like about your post is the fact that you are not holding back on the issues that arise when trying to predict market prices. As you rightly say the underlying model can only be used to illustrate the walk forward part of that process.

You will find many blog posts claiming to predict market prices when all they actually do is creating a noisy t-1 that can't beat the t-1 benchmark. This isn't surprising because the model will learn that if its predicted prices are just around the area of yesterday's price it will be good.

Despite looking like the easier problem to solve, trying to predict if today's close is above or below yesterday's close might have more practical relevance.

As you said, I wanted to make cristal clear that the post was aimed to explain the walk-forward analysis, without any machine learning model in particular.

About differencing the data for classification:

KlondikeFX:

Despite looking like the easier problem to solve, trying to predict if today's close is above or below yesterday's close might have more practical relevance.

You are right, predicting if today's close will be positive or negative would be a better approach to the problem, and it also allows you to implement classification models, not only the regressions ones.

In the post I made a reference to differencing the data;

TradingMove:

Note that if instead of using the raw data, you want to use the use the difference from step ‘t’ to step ‘t-1’, you should insert the following lines before ‘return data’ :

data = data.diff()data = data.tail(len(data)-1)

This way you could later build a binary class and apply classification. But I choose plain regression without differencing because I thought the graph with the results would be more explanatory, and as I said the post is aimed to explain the walk-forward analysis, but you are right. This is the graph with differencing.

Another important topic I want to address is the introduction of the Quandl API. Machine learning models are just algorithms that try to minimize the error between the true data and the predicted data. In order to improve the predictions, more representative data should be supplied to the model, and Quandl is a great place to start finding data.

If I find that many people is interested in machine learning with python I will try to post more often, and also I would love people posting about it. Maybe in a few months we are writting about Capsule nets and the best way to implement them to find candlestics patterns using sequential images as if they were a video of the market Which is a nice topic indeed...

@fiber100, you raise what is probably the most important consideration when using walk forward optimization. I've looked into this in some detail and the answer depends on the frequency at which your system trades. If you choose durations that are too short, your results for each in-sample/out-of-sample phase will not hold any statistical significance. However, choose periods that are too long and you will not benefit from the ability of walk forward optimization to illustrate how well your system is able to adapt to changing market conditions. You have to get the balance right. If you are interested I have written a 4 part blog post on getting this balance right to ensure that you get the most from WFO. You can find part 1 here: http://www.tradelikeamachine.com/blog/making-walk-forward-analysis-really-work-for-you-part-1

These are 1 million dollar questions. There are a lot of guidelines that make sense but reality is that every strategy is a world apart. WFO is a tool like many others.Optimizing the right way is the skill of the algorithmic trader.

@TradeSignalMachine thanks for the link, "balancing act" idea was new to me. Also I think after finding strategy working well on particular WFO period, retesting it on slight variations of WFO periods should indicate robustness of strategy. Am i right? Of course this should be used on long enough time periods to generate significant number of trades when changing number of periods do not affect balance of statistical significance. Also I noticed that sometimes strategy shows good result on particular WFO period and also on period multiplied or divided by 2.

I suspect it could be because of market cycles and looking for programming tools to check it. What can be used for detecting cycles? Maybe Hilbert transform ? Then idea would be to plan WFO cycles accordingly to market cycles.

Nice post, but i doubt that WFO is the key. Since in Zorro it's very easy to do (just a few lines of parameters, simple even for me) i was experimenting a bit, and it never was better then just 1 robust, fixed parameter over all time. I like to start the backtest at a point where the price was the same as now (if possible), including all markets (bullish, bearish, sideways) and there you go.

With WFO it's like trying to catch a rabbit, everytime it changes the direction quick, you follow with a delay, but then it changes again...

Hi easyFX, if I am not misunderstanding you, if you are using all your data to optimize your strategy, and the results are better than an anchored walkforward with enough "n_samples", it's likely that you are overfitting.

What i do is some out of sample tests. Best is if the parameter is very robust (close values are nearly as good, so no spike of a particular value) and in out of sample test it should be similar. I also test it with some older data. My system would have worked since beginning of my data, which is 1987... with DM before Euro (not counting, that it was not possible at that time to trade this, due to high spreads and limited possibility to trade)

CFDs are complex instruments and come with a high risk of losing money rapidly due to leverage. 73 % of retail investor accounts lose money when trading CFDs with this provider. You should consider whether you understand how CFDs work and whether you can afford to take the high risk of losing your money.