Discussions about the testing and simulation of mechanical trading systems using historical data and other methods. Trading Blox Customers should post Trading Blox specific questions in the Customer Support forum.

I was recently attempting to describe in concrete terms what an "edge" was. For example, what is the "edge" for a particular entry. In particular, I wanted to break apart the Donchian system, a simpler version of the Turtle system which we supply with Trading Blox. The Donchian System's entries are composed of two parts, breakouts and a moving average crossover filter which only allows trades in the direction of the major trend.

I wanted to come up with a way of measuring the "edge" of the individual components of the entry separately.

I think several traders here have been exploring the concepts of random entries to determine if there are edges in their exits. For determining the edge for entries one could try random exits but another approach which works for this type of analysis is a simple time-based exit without stops. For example, exit the market after 70 days.

When thinking about the meaning of an "edge" for entries it seems to me that we are intested in the basic price movement subsequent to the entry. In particular we are interested in the ratio of the adverse price movement to the favorable price movement.

For example, if you have a buy entry signal you want to see how much the subsequent price moves up as compared with down; how much it moves in a direction favorable to your trade as compared with how much it moves in a direction adverse to your trade.

As part of my research, I hypothesized that an entry with an edge will have a ratio of favorable price movement to adverse price movement that is significantly better than 1.0, while a random entry should have something close to 1.0.

I developed a new ratio (new to me at least) I call the Excursion Ratio which is defined as:

1) Computing the MFE and MAE for each trade.
2) Dividing each by the ATR at entry to adjust for volatility and normalize across markets
3) Summing these separately and dividing by the number of trades to get the average
4) Dividing Average Volatility-Adjusted MFE by Volatility-Adjusted MAE

I don't know if this is unique or not but it seemed to me like an obvious way to measure and compare the edge of an entry.

I used the Excursion Ratio to examine the edge for various values for the number of days for the high and low channel for the breakout using a simple 70 day exit. The following graph shows Excursion Ratio 70 plotted for values varying from 10 to 100:

These results fit well with my hypothesis that an edge would exhibit an Excursion Ratio greater than 1.0. It is also an interesting confirmation that the old timeframes from Turtle days of 10 week breakouts still represent the peak edge when compared to breakouts of shorter and longer length.

To further test the utility of the measure, I ran 10 tests where the same 70 day exit was combined with a random entry on the open.
For 10 tests the Excursion Ratio 70 varied from 0.89 to 1.16 centered very close to the 1.0 which one might have predicted. This too confirmed my initial hypothesis that market prices after random entries would exhibit no tendency to move in any particular direction.

I wanted to use this new tool to see how the different components of the edge broke out for the Donchian System. In particular, how the MACD filter added an edge. Interestingly the addition of the MACD filter improves the results of both random and breakout entries.

The following shows the Excursion Ratio 70 of breakouts when using the MACD filter:

Notice how the Excursion Ratio improved for all of the values of days in in the breakout.

I also tested random entry made using the MACD filter using a 300 day and a 50 day MA. The ten Excursion Ratio 70 values ranged from a low of 0.96 to a high of 1.48 with an average of 1.24. This indicates to me that a simple trend filter provides an edge and that anything you do in the direction of the trend will create further edge.

After satisfying myself with the utility of the Excursion Ratio as a measure of what I intended I wanted to further explore the edge provided by a breakout. In particular, I held the hypothesis that the edge would change as one approached the price for a breakout.

In Trading Blox there is a parameter which comes as part of the Donchian System called the Entry Offset which lets you change the relationship of the entry stop to the actual breakout itself in terms of the Average True Range. A negative Entry Offset of 1.0 means enter a stop 1.0 ATR before the breakout, a positive Entry Offset of 1.0 means 1.0 ATR past the breakout price.

I ran a test varying the entry Offset from -1.5 to 1.5 ATR and the following graph shows how the ER 70 values change across that range:

This graph fit my expectations pretty well. The edge increases for the breakout stop entry until just before the breakout when it levels off and then descends a bit at the end. This would tend to indicate that the best time to enter would be just ahead of the breakout which fits pretty well with my experience.

I was pretty satisfied with the measure when I realized that I had left the MACD Filter on. This piqued my curiousity. I wondered what the edge would look like if one turned the MACD off and looked at all breakouts, not just those that were in the direction of the trend.

So I ran another test using the exact same parameters with the exception of the MACD filter being turned off:

To be quite frank, I was not expecting this graph. You might even say I was shocked by this graph, at first at least.

As I began to think about it I realized that it made perfect sense. If you trade with the trend a breakout has positive edge. If you discount the trend then the breakout appears to have less edge than the prices either before or after. This fits very well with my perception that prices will either tend to bounce off breakouts and reverse or go through. So the fact that a price has gotten to a breakout is not that meaningful if you don't consider the general trend. It also means that a price which is say 1.0 ATR from a breakout is fairly likely to reach the price of the breakout.

Out of curiousity, I wanted to isolate the critical trades which were not part of the initial test. In other words, I wanted to isolate out the 180 or so trades from the second test which had been against the MACD filter. So I opened up the MACD Filter Block and added a new parameter to reverse the MACD and ran the same test with this new reverse MACD filter.

The interesting thing is that all of the values for this graph show negative edge. Taking a breakout against the major trend results in price movement that moves more against your trade than in the direction of the trade.

This is not surprising and does certainly explain why excluding the MACD Filter from the second test resulted in much lower values for edge and for the V-shaped ER 70 curve.

What are you using to measure the "edge" of entries and exits?

Has anyone seen anything similar to or exactly the same as the excursion ratio before?

- Forum Mgmnt

Trading Blox Users (Pro and Builder Only)
--------------------------------------------------------
The Blox for these tests are available in the Blox MarketPlace at:viewtopic.php?t=3317

Forum Mgmnt wrote:Has anyone seen anything similar to or exactly the same as the excursion ratio before?

Forum Mgmnt, you might want to have a look at Chapters 8 and 9 of Thomas Stridsman's book Trading Systems That Work. He goes over a large number of variations on the theme of Entry Efficiency. (Most of them involve "drawdown" so they are certainly not identical to your Excursion Ratio which doesn't include drawdown).

Of greater interest is Stridsman's discussion of the number "Entry Efficiency" calculated and printed by Tradestation and its software add-in partner RINA Systems. For a long trade,

As you can see, the numerator is the MFE, just as in your Efficiency Ratio. The denominator is not exactly the MAE, it's worse**, but it's awfully similar. For a single trade, their Efficiency and your Efficiency are awfully similar.

TS/RINA, however, compute this Efficiency ratio for each individual trade, and then compute the average of the ratios. Your method computes the ratio of the averages (sum the numerators, sum the denominators, divide the sums) which is different. Also your idea to normalize both MAE and MFE by entry volatility is (as far as I'm aware) brand new. Contratulations!

Another book to compare against is Perry Kaufman's Smarter Trading. He defines an Efficiency Ratio (pp. 134-152) which can be applied to a trade. For a long,

Forum Mgmnt wrote:I don't know if this is unique or not but it seemed to me like an obvious way to measure and compare the edge of an entry.

I still think you are testing the efficiency of a complete system. It's just that it is a system that has a time based exit .

I spent a lot of time on trying to grasp "edge testing" on individual components some time ago (for example see here) but I finally gave it up; all parts of a system seem interrelated and the sum of the parts might not be the same as the whole. And one doesn't know in what direction an individual component will push the sum of the whole.

tobbe wrote:I still think you are testing the efficiency of a complete system

You are correct. It really is a system. It has the advantage of being one where the exits are not based on price action, however.

In fact, if you use the Multi-Money Manager's "Volatilty Adjusted" position sizing which sizes positions based on ATR, you get decent performance. The system is also very unlikely to have the same drawdown profile as a system that exits using a signal that is based on price action, so it is a good candidate for adding to a mix of other systems.

The following graph shows the equity curve for a system that does not have any stops, uses the above described entry and exits at the end of 80 days, and uses 0.5% per ATR position sizing. This is for a suboptimal portfolio which does not include any foreign markets since the sample data doesn't either.

Not too bad for a simple system with an even simpler exit.

NOTE: This system did not have a bad May like most trend-following systems with conventional exits.

sluggo wrote:For a single trade, their Efficiency and your Efficiency are awfully similar.
...
Also your idea to normalize both MAE and MFE by entry volatility is (as far as I'm aware) brand new. Contratulations!

sluggo,

Thank you for your comments.

I agree that the single trade efficiency and MFE/MAE are very similar. I thought it made sense to build upon concepts which were already well undstood and MFE and MAE are both well understood measures of the good and bad components of a trade.

With respect to normalizing for volatility, it is interesting to see how much our thinking is constrained by the available tools for testing and measurement. When everyone is used to thinking in terms of a single market, which is very easy to do when the tool you are most comfortable with is TradeStation, you find solutions that work fine in those environments.

I happened to have been educated to think of normalizing across markets and differing levels of volatility from the early Turtle days so to me this was just an obvious step. I'm interested in behavior across markets so naturally I will use a tool which I have found very useful for such things in other areas, volatility and specifically ATR as a measure thereof. If one is going to use stops or position sizing that is volatility adjusted it seems imperative that any measures also be normalized for volatilty.

Averaging the individual components and then taking a ratio (versus averaging the ratios) has the advantage of increasing the importance of outliers in either direction. The other approach works but is less sensitive to outliers.

In an earlier post on this topic, I wrote:To further test the utility of the measure, I ran 10 tests where the same 70 day exit was combined with a random entry on the open.

For 10 tests the Excursion Ratio 70 varied from 0.89 to 1.16 centered very close to the 1.0 which one might have predicted. This too confirmed my initial hypothesis that market prices after random entries would exhibit no tendency to move in any particular direction.

It turns out I was completely wrong here. I ran a short 10-test simulation where this appeared to be true but this just goes to illustrate that small sample sizes can be dangerous.

Random Entries do not have an E70-Ratio (short for 70 day Excursion Ratio) of 1.0, it is more like 0.88. The values for shorter timeframes are even lower, for example at 10 days, the E10-Ratio is 0.73, the E5-Ratio is 0.64. These measure are averages of 100 different 10 year tests over 28 different markets.

I am going to leave it as an excercise for the reader to try and figure out why that might be for a few days.

For those of you without Trading Blox Builder, The algorithm I'm using for doing random entries is:

1) Pick a random number between 1 and 2 inclusive
2) If it is 1 go Long on the Open, if it is 2 go Short on the Open
3) Exit at the defined number of days on the Close
4) Wait a random number of days between 1 and 100 before entering the market again.

The implications of these numbers are interesting as it means that the edges delivered by the breakouts and MACD Filter are comparatively higher than I had previously thought, i.e. they add even more edge than I had thought.

I would like to add my humble two cents to this thread and also ask you some questions.

IMHO, I believe it is important to evaluate not only how much "edge" our entries may have, but also, how fast it may manifest. Personally, I would choose an entry or parameter set that gives me profits as fast as possible. I think this is important to evaluate how much "pain" one would have to endure before seeing a profit. I think this can also be useful for setting time-stops and optimizing capital efficiency across a large portfolio.

I have been playing with these blox since you released them, and have stumbled upon some puzzling results.

I ran a simulation with the donchian entry block and entry edge tester, were I stepped the "entry breakout" and "exit bars" parameters. The result was the first scatterplot below.

Nothing new there. But after seeing it, the trade charts, and the entry code, I noticed that this "system" may only have a single long or short position at any time. It may have two positions, a long and a short one, but not multiple long positions. So I wondered: "And what if I take each and every signal generated by the entry block? After all, we are testing the entry edge, not the system, right?". So, I modified the code to take multiple positions and re-ran the test over the same time period and portfolio. The results surprised me, and can be seen at the second scatterplot.

The amount of "edge" for each area of the chart increased (about 70%), and the "optimal" region moved from one side to the other. I still donÂ´t know what to make of this. I didnÂ´t expect such a change. I think my test procedure is correct, as it yields a much bigger sample (almost ten times bigger) and a more evenly distributed chart, but at this moment, I canÂ´t stop but question it.

So, here is my question: what is your opinion on this matter? Am I missing something here? Am I plain wrong or what? I understand that there is no such thing as absolute right or wrong in this subject, but in this case I really need the opinion of someone more experienced.

So, here is my question: what is your opinion on this matter? Am I missing something here? Am I plain wrong or what? I understand that there is no such thing as absolute right or wrong in this subject, but in this case I really need the opinion of someone more experienced.

Thanks in advance.

I have noticed that portfolio composition is a big factor on where the edge lies strongest.

If one is trading the all liquid portfolio, should all testing be done only on the all liquid portfolio? Or should testing be performed on the widest selection of markets possible in an effort to increase robustness? (at the possible expence of weaker performance in a backtest on all liquid in isolation)

Forum Mgmnt wrote:
It turns out I was completely wrong here. I ran a short 10-test simulation where this appeared to be true but this just goes to illustrate that small sample sizes can be dangerous.

Random Entries do not have an E70-Ratio (short for 70 day Excursion Ratio) of 1.0, it is more like 0.88. The values for shorter timeframes are even lower, for example at 10 days, the E10-Ratio is 0.73, the E5-Ratio is 0.64. These measure are averages of 100 different 10 year tests over 28 different markets.

I am going to leave it as an excercise for the reader to try and figure out why that might be for a few days.

- Forum Mgmnt

I was quite certain that the Excursion Ratio difference between longer and shorter hold timeframes was entirely due to transaction and market impact costs but setting all costs to zero still consistently results in E5-Ratio < E70-Ratio.

In fact, the difference between E70-Ratio and E5-Ratio hovers around 0.20 - most definitely a statistically significant difference! I must be overlooking something basic - please enlighten us c.f..

1) Pick a random number between 1 and 2 inclusive
2) If it is 1 go Long on the Open, if it is 2 go Short on the Open
3) Exit at the defined number of days on the Close
4) Wait a random number of days between 1 and 100 before entering the market again.

May not always make sense (and I'm still a beginner after years) but this is what I do to try not to lose the "system context" when testing entry/exit only:
-keep entry rule fixed when testing exit and viceversa
-run several '000 backtests entering (exiting) at random
-adjust the probability of entering(exiting) at every run trying to obtain the same number of trades as in the complete system for the same period (keep cost, # of opportunities constant)
-compare average % return with original system % return (no money mgmt)

In an earlier post, I wrote:Random Entries do not have an Excursion Ratio of 1.0, it is more like 0.88...

...the values for shorter timeframes are even lower...

...I am going to leave it as an excercise for the reader to try and figure out why this might be for a few days.

This was a bit of a trick quesion.

The short answer is that this should NOT happen. There must be something going on that we don't understand. If you really have random trade direction, i.e. a random trade is as likely to be long and short then there should be no tendency to move for or against a position. Even if there is a tendency for the market to move higher (and there is), one would expect this tendency not to show up in the Excursion Ratio measurements for random entries that include both long and short trades.

This often happens while testing, but whenever I get an answer I don't expect it forces me to dig deeper until I do understand those answers. In this case I investigated by testing a short time period and then by inverting the logic behind the long and short trade entries. I then compared the MAE and MFE for both sets of trades and they were not the inverse as I had expected.

The reasons for these were:

1) The effect of Slippage calculations. If you factor some slippage in, it will always increase your adverse excursion values and decrease your favorable excursion values. If you consider two trades for the same length on the same market starting on the same day but in opposite directions you will see that if one considers price movement after the fill, the MAE and MFE for the two trades are not the inverse as one might intuitively expect.

Since the price movements for trades will be larger for a larger number of days of measurement, the shorter intervals are more greatly affected by this effect.

2) There was an error in the calculation for the MAE on the day of exit for short positions. This is one of the reasons that shorter values had a larger error. The error of a single day would be relatively larger for shorter exit periods. This problem was fixed in version 2.1.11 which was released on Friday. This error occurred less frequently but it did affect the results.

So with version 2.1.11, if one sets the slippage to zero one will indeed get a 1.0 Excursion Ratio for random entries at any length of measurement.

I would like to take a stand at the first contribution of this thread, written by c.f.:

When thinking about the meaning of an "edge" for entries it seems to me that we are intested in the basic price movement subsequent to the entry. In particular we are interested in the ratio of the adverse price movement to the favorable price movement.

c.f.; I have thought over this. An other concept would be to evaluate the max. potential of an Entry-method, i.e. I always suppose an optimal Exit for each trade.The advantage of such a method is the following: In a further step of the system development process you will have to find an good Exit-Method. If you know the max. potential of your Entry method, you can judge, how good your exit method is.

So the formula for the Entry Efficiency which you suggested in the first contribution was:

When I suppose an optimal Exit method, then the Exit will always be at the bar with the maximum favorable excursion. So, if there is a new lowest low after this bar, it does not matter:

Example1: For long-trades: If the bar which is responsible for the "maximum adverse excursion" is generated after the bar which is responsible for the "Maximum Favorable Excursion", then you would have sold the instrument (stock) at the bar which was responsible for the "Maximum Favorable Excursion" to get the max. profit.

Example2: For long-trades: If the bar which is responsible for the "maximum adverse excursion" is generated before the bar which is responsible for the "Maximum Favorable Excursion", then there is no difference between your ratio and my ratio.

So I would suggest the following variation to get the optimal Excursion Ratio:

Average Volatility-Adjusted MFE
------------------------------------------------------------------------------------
Average Volatility-Adjusted MAE between the Entry and the bar with the MFE

Please have a look at the attached picture for better comprehension.

Maybe a disadvantage of my method is the following:
In your "Excursion Ratio", a value of 1.0 means, that the Entry-method is not better than the coincidence, but in my "optimal Excursion Ratio" a randomly Entry performs also a "optimal Excursion Ratio" which value is higher than 1.0. But is this really a disadvantage? I mean, once you know the "optimal Excursion Ratio"-value for the coincidence there is no problem.

I think an advantage of my method is that after adding an Exit method in a further process of the system developing process, you can exactly see, how much of the entry potential your Exit-methode can exploit.

What do you think about this? All contributions are welcome. Thank you.