Monday, April 13, 2015

(This post is based on the talk of the same title I gave at Quantopian's NYC conference which commenced at 3.14.15 9:26:54. Do these numbers remind you of something?)

A correct backtest of a trading strategy requires accurate historical data. This isn't controversial. Historical data that is full of errors will generate fictitious profits for mean-reverting strategies, since noise in prices is mean-reverting. However, what is lesser known is how perfectly accurate capture of historical prices, if done in a sub-optimal way, can still lead to dangerously inflated backtest results. I will illustrate this with three simple strategies.

CEF Premum Reversion

Patro et al published a paper on trading the mean reversion of closed-end funds’ (CEF) premium. Based on rational analysis, the market value of a CEF should be the same as the net asset value (NAV) of its holdings. So the strategy to exploit any differences is both reasonable and simple: rank all the CEF's by their % difference ("premium") between market value and NAV, and short the quintile with the highest premium and buy the quintile with the lowest (maybe negative) premium. Hold them for a month, and repeat. (You can try this on a daily basis too, since Bloomberg provides daily NAV data.) The Sharpe ratio of this strategy from 1998-2011 is 1.5. Transaction costs are ignored, but shouldn't be significant for a monthly rebalance strategy.

The authors are irreproachable for their use of high quality price data provided by CRSP and monthly fund NAV data from Bloomberg for their backtest. So I was quite confident that I can reproduce their results with the same data from CRSP, and with historical NAV data from Compustat instead. Indeed, here is the cumulative returns chart from my own backtest (click to enlarge):

However, I also know that there is one detail that many traders and academic researchers neglect when they backtest daily strategies for stocks, ETFs, or CEFs. They often use the "consolidated" closing price as the execution price, instead of the "official" (also called "auction" or "primary") closing price. To understand the difference, one has to remember that the US stock market is a network of over 60 "market centers" (see the teaching notes of Prof. Joel Hasbrouck for an excellent review of the US stock market structure). The exact price at which one's order will be executed is highly dependent on the exact market center to which it has been routed. A natural way to execute this CEF strategy is to send a market-on-close (MOC) or limit-on-close (LOC) order near the close, since this is the way we can participate in the closing auction and avoid paying the bid-ask spread. Such orders will be routed to the primary exchange for each stock, ETF, or CEF, and the price it is filled at will be the official/auction/primary price at that exchange. On the other hand, the price that most free data service (such as Yahoo Finance) provides is the consolidated price, which is merely that of the last transaction received by the Securities Information Processor (SIP) from any one of these market centers on or before 4pm ET. There is no reason to believe that one's order will be routed to that particular market center and was executed at that price at all. Unfortunately, the CEF strategy was tested on this consolidated price. So I decide to backtest it again with the official closing price.

Where can we find historical official closing price? Bloomberg provides that, but it is an expensive subscription. CRSP data has conveniently included the last bid and ask that can be used to compute the mid price at 4pm which is a good estimate of the official closing price. This mid price is what I used for a revised backtest. But the CRSP data also doesn't come cheap - I only used it because my academic affiliation allowed me free access. There is, however, an unexpected source that does provide the official closing price at a reasonable rate: QuantGo.com will rent us tick data that has a Cross flag for the closing auction trade. How ironic: the cheapest way to properly backtest a strategy that trades only once a month requires tick data time-stamped at 1 millisecond, with special tags for each trade!

So what is the cumulative returns using the mid price for our backtest?

Opening Gap Reversion
Readers of my book will be familiar with this strategy (Example 4.1): start with the SPX universe, buy the 10 stocks that gapped down most at the open, and short the 10 that gapped up most. Liquidate everything at the close. We can apply various technical or fundamental filters to make this strategy more robust, but the essential driver of the returns is mean-reversion of the overnight gap (i.e. reversion of the return from the previous close to today's open).

We have backtested this strategy using the closing mid price as I recommended above, and including a further 5 bps transaction cost each for the entry and exit trade. The backtest looked wonderful, so we traded it live. Here is the comparison of the backtest vs live cumulative P&L:

Yes, it is still mildly profitable, but nowhere near the profitability of the backtest, or more precisely, walk-forward test. What went wrong? Two things:

Just like the closing price, we should have used the official/auction/primary open price. Unfortunately CRSP does not provide the opening bid-ask, so we couldn't have estimated the open price from the mid price. QuantGo, though, does provide a Cross flag for the opening auction trade as well.

To generate the limit on open (LOO) or market on open (MOO) orders suitable for executing this strategy, we need to submit the order using the pre-market quotes before 9:28am ET, based on Nasdaq's rules.

Once again, a strategy that is seemingly low frequency, with just an entry at the open and an exit at the close, actually requires TAQ (ticks and quotes) data to backtest properly.

Futures Momentum

Lest you think that this requirement for TAQ data for backtesting only applies to mean reversion strategies, we can consider the following futures momentum strategy that can be applied to the gasoline (RB), gold (GC), or various other contracts trading on the NYMEX.

At the end of a trading session (defined as the previous day's open outcry close to today's open outcry close), rank all the trades or quotes in that session. We buy a contract in the next session if the last price is above the 95th percentile, sell it if it drops below the 60th (this serves as a stop loss). Similarly, we short a contract if the last price is below the 5th percentile, and buy cover if it goes above the 40th.

Despite being an intraday strategy, it typically trades only 1 roundtrip a day - a low frequency strategy. We backtested it two ways: with 1-min trade bars (prices are from back-adjusted continuous contracts provided by eSignal), and with best bid-offer (BBO) quotes with 1 ms time stamps (from QuantGo's actual contract prices, not backadjusted).

For all the contracts that we have tested, the 1-ms data produced much worse returns than the 1-min data. The reason is interesting: 1-ms data shows that the strategy exhibits high frequency flip-flops. These are sudden changes in the order book (in particular, BBO quotes) that quickly revert. Some observers have called these flip-flops "mini flash crashes", and they happen as frequently in the futures as in the stock market, and occasionally in the spot Forex market as well. Some people have blamed it on high frequency traders. But I think flip-flop describe the situation better than flash crash, since flash crash implies the sudden disappearance of quotes or liquidity from the order book, while in a flip-flopping situation, new quotes/liquidity above the BBO can suddenly appear and disappear in a few milliseconds, simultaneous with the disappearance and re-appearance of quotes on the opposite side of the order book. Since ours is a momentum strategy, such reversals of course create losses. These losses are very real, and we experienced it in live trading. But these losses are also undetectable if we backtest using 1-min bar data.

Some readers may object: if the 1-min bar backtest shows good profits, why not just trade this live with 1-min bar data and preserve its profit? Let's consider why this doesn't actually allow us to avoid using TAQ data. Note that we were able to avoid the flip-flops using 1-min data only because we were lucky in our backtest - it wasn't because we had some trading rule that prevented our entering or exiting a position when the flip-flops occurred. How then are we to ensure that our luck will continue with live market data? At the very least, we have to test this strategy with many sets of 1-min bar data, and choose the set that shows the worst returns as part of our stress testing. For example, one set may be [9:00:00, 9:01:00, 9:02:00, ...,] and the second set may be [9:00:00.001, 9:01:00.001, 9:02:00.001, ...], etc. This backtest, however, still requires TAQ data, since no historical data vendor I know of provides such multiple sets of time-shifted bars!

As I mentioned above, these flip-flops are omnipresent in the stock market as well. This shouldn't be surprising considering that 50% of the stock transaction volume is due to high frequency trading. It is particularly damaging when we are trading spreads, such as the ETF pair EWA vs EWC. A small change in the BBO of a leg may represent a big percentage change in the spread, which itself may be just a few ticks wide. So such flip-flops can frequently trigger orders which are filled at much worse prices than expected.

Conclusion

The three example strategies above illustrate that even when a strategy trades at low frequency, maybe as low as once a month, we often still require high frequency TAQ data to backtest it properly, or even economically. If the strategy trades intraday, even if just once a day, then this requirement becomes all the more important due to the flip-flopping of the order book in the millisecond time frame.

130 comments:

Anonymous
said...

You are making general inductions from two examples that represent noise trading. naturally there would be affected by noise in closing prices. But you conclusions do not apply to all strategies. Look at the cumulative return of the first strategy: it is only +0.8%. This is noise and will be affected by small changes in closing values. The same holds for the second strategy: 0.35%. It is not the closing price, it is the strategies you are testing that have the problems and the data just tell you that. I challenge you to do the same for a moving average cross over system in S&P 500. Will you get any significant differences? The conclusions from your article is about the strategies, not about the data because the future can be different than the past. In other words the conclusion should be: Beware of noise trading.

I am afraid you have misinterpreted the cumulative returns of the first strategy: it is over 60%, not 0.8%.

Also, the third strategy is given especially in anticipation of some readers who would object that this phenomenon only applies to mean-reversion strategies ("noise-trading" in your parlance). That is a momentum strategy, and if it were to rely only on noise, it would not have generated any positive returns when backtested with 1-min data. Noise kills that strategy, but noise is not the profit driver.

We trade live a highly profitable mean-reverting strategy, so I believe "Beware of Noise Trading" would not be an accurate portrayal.

Yes Sir, 60%, sorry for the typo, but in 7+ years, averaging less than 10% return. This can be noise trading. You should be able to find many such strategies that are sensitive to variations in the closing price. You are committing to a logical fallacy here of generalization. You should not trade any strategy that is sensitive to noise in execution price. Actually what you have proven is that some strategies are sensitive to variation in prices. You have not proven that ALL strategies are sensitive and that high freq data are required to test them. Do you think that a strategy that trades twice a week QQQ with about $2 PT/ST should be back-tested on millisecond data? Please if you think so, say it.

My article is not a mathematical proof. I do not claim that ALL strategies require 1-millisecond data. I said you should carefully consider if yours does require such data.

The CEF strategy trades only once a month, yet it requires 1-min data to backtest economically, unless you have expensive Bloomberg or CRSP subscription. So yes, if you trade QQQ twice a week at the close, you do actually need 1-ms data to backtest without Bloomberg/CRSP.

Ernie, thanks for this post. Would you mind posting some performance figures (Sharpe etc) on the intraday futures momentum strategy you mentioned? I ran this on 35 commodities over 10 years using minutely data and I'm afraid I do not get anything remotely looking like a good strategy - at least with these parameters. In fact, some commodities such as Beans likely show opposing behaviour, whereas for other commodities (RB, NG, CL, etc) it appears to be mostly noise. Are there any details that I am missing? Let me know - If you prefer, I can also contact you via email.

What kind of execution lag are you using? I would guess a strategy that is robust to a full day lag would be more robust to this sort of error than one that is only robust without a trading lag. E.g., how would the strategies you showed work if there was a 1-day lag, but still using daily data?

If your trading is successful why do you need to travel around the world and why do you need to insert affiliate links in this post to make commissions from the vendor selling the HF data? I know people who are trading and have no time to go for a walk while you travel constantly to give seminars. As far as this post, nothing is clear, nobody can confirm your findings and claims. Also, the strategies that execute at the close try mostly to arbitrage close price anomalies and are naturally very sensitive to variations in price. I would think a batter idea would be to avoid such strategies.

I am sorry to hear that your trader friends are unable to take walks while trading. I take about 2 hours of walks a day. Furthermore, I enjoy travelling around the world giving talks and seminars. I think Milan and London are both delightful cities to visit and I very much look forward to meeting traders there. After all this walking and travel, I still have time to perform research for our fund and teaching at the Master of Predictive Analytics program at Northwestern University. The reason simply is that our systems are fully automated, and require little human intervention. (Our team does get notified of systems failures by text messages though.)

If you would like to look at our trading track records, please visit epchan.com/accounts. We can provide brokerage statements to prospective investors with serious interest. Our commodity pool's track record is audited and filed with the NFA, and also available for your review on request.

If you believe that successful fund managers should forgo revenue generated by their blogs, seminars, or books, you might wish to also contact best-selling author and fund manager Mr. Joel Greenblatt. He manages multi-billion dollars of assets, so by your logic he should give away his books free of charge.

Regarding the strategies described in this post, everyone can verify if my conclusions are correct or not. The strategies are fully described either here or in my second book or in the cited and published journal article. Please feel free to backtest using consolidated EOD data vs official close and see the differences for yourself.

Finally, regarding QuantGo: yes, I think their business is great and I do wish to support them by providing links. Our fund is a customer and we want them to prosper so that we can continue to benefit from their service. But you don't need to use QuantGo at all to backtest the first two strategies. You can subscribe to Bloomberg. For the third strategy, you can buy 1-ms data from tickdata.com instead. In a similar vein, I support all new businesses that enhance trader productivity such as Quantopian or Rizm or Marketcetera etc., to all of whom I have provided links previously.

Hi Ernie, thanks for your post, I am concerned about the point regarding the 3rd issue. What was the percentage difference between the returns shown by per minute data, and those for live (or for with per ms data).

I have generally been concerned with backtest results - but I continuously check actual broker execution prices against the data provider's prices which are per minute, and the inevitable difference is something I consider part of the overall transaction cost - the difference being a kind of slippage. The cost, though not negligible, is not that high. Have you used per minute prices for your mean reversion strategies in the past?

Also, for the flip flops case, was there a need felt to have minimal difference between the 95 percentile price and the last price etc.?

While I respect your desire for privacy when making anonymous comment, I do suggest that you sign it with your first name, or a nickname. That will make reading a Q&A thread that much easier for everyone.

No, I don't think entering a few ticks above 95th percentile will change anything. Please note that the strategy does not exit whenever the price drops below 95th: it only exits when it drops below 60th. So the flip-flop is not a matter of just a few ticks' changes. Hence the name "mini flash crash". The change of just a few ticks won't constitute a crash.

Hi Ernie. Thanks for the insightful, practical-oriented books, blog, and also some of your excellent comebacks in this particular thread... While the gotchas in the first two strategies are intuitive, I find the futures momentum one a bit surprising. Instead of using quotes or 1 min bar trades, couldn't you just use all trades (all ticks), excluding quotes? The strategy would be exactly the same, but you would have to test using either a) a fixed cost measure for implementation shortfall, b) execution lag (i.e. trade on lastpx at T+50ms) or c) both. I find quote data has inherently lower quality and higher degree of errors, and I would be very hesitant to use it in a strategy without proper cleaning and validation. Depending on the alpha decay in the strategy (shouldn't be too bad for this one), various clever execution approaches can minimise the total cost (slippage, opp. cost) and providing a small and fairly deterministic (over time) tracking error to sim.

Yes, we can rank trades instead of quotes, but the result won't be very different. And indeed, smarter execution strategy with a lag may help avoid some instances of flip-flops, but not all. The key problem with this strategy is not slippage, but the sudden move and reversal of the BBO and the corresponding trades.

Most FX trading strategies apply higher leverage than is customary for equity strategies because the daily movement of currency rates for major pairs seldom exceeds 1%, and never exceeds 3%. (We do not trade pairs that are pegged by central banks, such as EURCHF or USDHKD). So even at 10x leverage, 3% loss translates to 30%, hardly a "total loss" that you are concerned with. Besides, we have stop losses at much lower levels.

If you are a more risk-averse investor, you can choose a lower leverage for your managed account, such as 5x. Then 2014 return would be reduced to about 34%, and CAGR would be reduced to about 20%. This would still put us in the top quintile of hedge fund returns. (At this leverage, we will need a 20% move to suffer a "total loss". Realistically, if for e.g. EURUSD moves 20% in a day, large number of hedge funds will be out-of-business.)

Even more risk-averse investors who choose a 2.5x leverage will still enjoy nearly 10% annual return - hardly a dismal number.

If you do not wish to invest in any leveraged strategies at all, then please don't go near quantitative trading strategies or hedge funds. Investing in the SPY, with its sometimes decade-long drawdown duration, might be more suitable for your taste.

I've read your books and followed your blog for about a year. I get the impression (which may be inaccurate) that you've thrown a lot of strategies aside and are focusing on the forex strategy in your commodity pool, which looks like it has performed very well. I've tried a number of strategies, no holy grails, and the approach that seems to be taking shape for me is to find enough mediocre but still positive strategies so that the portfolio Sharpe is acceptable even if the individual Sharpes are shaky.

Do you have any thoughts on this philosophical question of the holy grail vs. diversification? That is, the approach of "most strategies are disappointing so keep looking and when you find one that works, pile into it" vs "most strategies are disappointing but you can combine them into a whole that is better than the parts (as long as they're at least positive)"?

Hi Brian,FX is indeed the most profitable strategy that we have traded, and that's why we are offering that to Managed Account clients. However, we do trade a variety of other futures and stocks strategies in our commodity pool for the purpose of diversification. The strategies in the pool are traded at much lower leverage than the one in the Managed Account. The pool is also performing well - up over 4% net in Q1.

Diversification of strategy is always a good idea. As you pointed out, uncorrelated strategies always increase the portfolio Sharpe ratio. Of course, one should continue to look for higher performing strategies at the same time. There is no contradiction between the two.

I read your books and have your blog on me feeder. I have to thank you for all the invaluable tips you have given us over the years. You have been more than generous with the information and I find it great that you not only share some of the things you know but make sure the little guy can try them too for a reasonable price.

I use moc orders from time to time and was quite surprised when you said the primary and consolidated closing price are different. What I have seen in my personal experience is that they are the same.

So I did a little test with a bloomberg terminal. I took the first 50 companies in the sp500 and got the closing prices for the last ten days. I not only got the prices from the primary exchange but the consolidated too and then I took the closing prices from google finance.

This means we had 100 comparisons (50 primary vs consolidated and 50 primary vs google finance) each day and for a total a 10 days that gives us 1000 comparisons.

Now I took the differences in prices and added everything together from both comparisons... results: $0.00.

I might be doing something wrong because it looks pretty simple and straightforward. Have you tried this in bloomberg? and if so what commands did you use?

Regards,Leo

p.s some times there are little differences, for instance if you check MSFT on the 15th on april you see that the closing price was 42.255 (http://www.nasdaq.com/symbol/msft/historical). That is on the nasdaq website and also in bloomberg but in google you see 42.26, probably because of rounding errors.

I have done research on the arbitrage between ETF and its components, but have not traded it live. I am concerned by the data-snooping bias that may be involved. See epchan.blogspot.com/2007/02/in-looking-for-pairs-of-financial.html

The spread is highly stock-dependent, even for "liquid" stocks. If measured by ticks, it will vary widely depending on stock price. So it is more sensible to measure it by basis points. 1 or 2 bps are common for SPX stocks.

Nav Sarao, the trader who allegedly "caused" the flash crash, is far from a HFT. By his own admissions, he point-and-click trades.

Limit order book in the stock market is, however, highly subject to manipulation. Half of my MFT course is about the different kinds of manipulation people do, and how to protect yourself against them.

If by "not making money from their traders" you mean they do not charge their traders monthly fees, exorbitant commissions, etc., then yes, there are such prop trading firms. In fact, most HFT shops belong to this category. But these firms only hire traders with demonstrated track records.

Millennium Partner operates almost like a fund of funds. They hire lots of traders, and everyone has some capital allocation and trade their own strategies as they see fit, (subject to risk constraints of course). No collaboration between traders are expected nor encouraged.

Many investors wondered what happened to our 2013 net return which, as you pointed out, were just 5% or so. The answer is that it has nothing to do with our strategy - it has to do with the entire FX market that year. Irrespective of currency pair, the volume and volatility of the market reached a low that is unseen in recent years. Large FX hedge funds such as FX Concepts went out of business, further draining liquidity. (See http://www.reuters.com/article/2013/12/19/us-forex-hedgefunds-idUSBRE9BI0P720131219). But just as suddenly, in mid-2014, volume returned, and all became normal again.

If your strategy is to trade once per month and the difference between consolidated and auction price is as reported on example above around 0.1%, thus having such a big difference in performance is a bit weird, isn't it? Is it due to the leverage?

In addition, does the bias between the two prices is correlated with strategies used? It seems to me that bias works always against the strategy.

Hi Jan,The example above used AAPL, the most liquid stocks. A typical CEF is far less liquid, and the difference between auction and consolidated price is likely far higher than 0.1%.

Furthermore, the CEF reversal strategy picked the outliers in returns to form the monthly portfolio. Hence it often picks those stocks with consolidated prices that are especially different from the auction prices. The average difference is not applicable here.

Hi Peter,It is not difficult to get a backtest with Sharpe ratio of 3, it is very difficult to get that in live trading over, say, 3 years, unless one is a high frequency trader.

Focusing on Sharpe ratio wholly overlooks the hidden tail risks. It preference mean reversion strategies over momentum strategies for no rational reason. It is, in short, a very unwise selection criterion. If a trading firm advertises such preference, you know it is run by amateurs.

Hi Ernie, Your last comment is interesting - if one did have a way of generating momentum strategies - any selection scheme even indirectly influenced by a Sharpe ratio will prefer mean reversion strategies over those. Can a mean/recent_drawdown estimate work better in this selection?

I typically prefer to have an economic story behind it, so that I know when it stops working, whether it is due to noise or whether the fundamentals have changed and I should just discontinue it.

However, I also hear Bob Mercer over at Renaissance Technologies said that one of his best strategies has no fundamental basis whatsoever, and because it seems so arbitrary, no one else trades it and therefore it remains profitable throughout the years.

A question regarding construction of strategies. It seems like if I make my strategy beta-neutral, it will not be zero cost. And if I make my strategy zero cost, it will not be beta neutral. How do I make my strategy both cash and beta neutral?

Hi Peter,Calmar ratio involves the calculation of cumulative compound returns. Compound returns are not proportional to leverage. The numerator and denominator do not involve the same compound returns. Hence the ratio is dependent on leverage.

Hi Samuel,Sharpe ratio measures risk-adjusted returns over and above the return on cash (i.e. cash is the benchmark). Hence we must subtract the risk-free-rate which is the return on cash. Equivalently, you can long the securities and short (borrow) the cash.

If your portfolio is long only, then you should use information ratio, which uses a different benchmark (SPX? NDX? Some bond index?) return which you should subtract from.

Hi Samuel,I should also clarify that if your portfolio is zero-cost, that means you don't need to short cash, and you won't need to subtract the risk-free rate from the returns in computing Sharpe ratio.Ernie

Hi Samuel,You can use Sharpe ratio for any portfolios. The only issue is whether the portfolio is zero-cost, in which case you don't need to subtract risk-free-rate from your returns. A futures portfolio is zero-cost, and risk free rate need not be subtracted.Ernie

In 1), returns are basically rolling three day returns for every new position taken.

In 2), daily returns are computed for a fixed weight vector even though you only realise the return after three days. I have ret(t+1:t+3) on the right hand side of the equality sign since there are overlapping positions taken every day.

In both ways, you are assuming that the capital allocation to each "stock" is constant - but unless you actively rebalance the positions at the end of each day, that won't be true. You need to specify shares instead of market value weights to compute returns.

The second way is more wrong because the (net) returns cannot be added - they need to be compounded like (1+r1)*(1+r2)-1. If you want to add, you have to use log returns instead.

As a starter I found that the biggest challenge is to find a reasonably cheap market data source, given that I have no profitable strategy to fund it initially. Thanks for the link I will check it out and see if it is applicable to myself.

Without the 1ms before close data, do you think we can use a probabilistic approach to estimate a worse case scenario?

Hi Paul,If you fund a small account at IB, they will give you free market data.You don't need to test with auction data in the beginning - just find a profitable strategy with consolidated data first and then verify with paper trading on IB.

I don't think you need to waste time with probabilistic approach to get around the auction data issue: it won't be right in any case.Ernie

If w_t represents the number of shares of a stock, then indeed the turnover is abs(w_t-w_t-1), but then you need to multiply by the price p_t to compute the turnover in $. If w_t represents market value, then this expression is incorrect.

Turnover is used to measure transaction cost. Your second expression is incorrect because the growth of market value due solely to market movement does not incur transaction cost.

Referring to your last paragraph. Say my signal suggests a portfolio weight of 0.60 for security i at time t-1. If security i has a higher return than your overall portfolio return over the period t-1:t then its weight has increased to say 0.65.

Now suppose your signal at time t+1 suggests a weight of 0.60 again. Then my turnover at t+1 using portfolio weights will be sum(abs(0.60-0.65)).

So in this case, the growth in the market value of the stock leads to turnover at t+1 for any signal weight different than 0.65.

Hi Samuel,In this case, the target weight is w(t-1)*(1+r_p), as you want your stock's weight to grow at same rate as the portfolio so its relative allocation is unchanged. But before rebalancing, the actual weight is w(t-1)*(1+r_i). (Assume throughout that w is the market value.)

Samuel,Maybe I don't understand fully what you are doing, or your notations.If a position has market value w(t-1), and its return from t-1 to t is r_i, how can it not be the case that the actual weight at t before rebalancing is w(t-1)*(1+r_i)?Ernie

You earlier alluded that it is not possible to make a portfolio both dollar neutral and beta neutral at the same time. Could you please expand on this? I don't see why, given that I have enough weights to satisfy the constraints.

Sorry for bothering you with yet another question but I find this stuff interesting :)

When constructing a continuous futures series, I know a popular method is backwards ratio adjustment. Just want make sure: There is no look-ahead bias arising from this right? I guess you only change the level of the entire series so returns should be unaffected. Is this correct reasoning?

Hi Sam,Backadjustment in futures means using the future price to adjust past prices: clearly there is a look-ahead bias. The way to remove this bias is to use the actual contracts to backtest, and manage the roll over to the next contract by buying/selling on the rollover day.Ernie

I want to hear your thoughts on neural networks. I know the quant-trading community has traditional dismissed NN as too black-box. But with recent developments in DNN, the field has furthered its understanding around network topology, alternative response functions, efficient ways to train large networks, etc.

Do you think this could potentially be a powerful tool for recognizing patterns from the financial market? And yes what type of frequenties/asset classes would it be most suitable?

I have been following your blog. I dont think there is a look ahead bias when using for example backwards ratio adjustment of futures. You just shift the level of the price up and down which does not matter if you work with returns.

If you back-adjust futures prices by eliminating the return gap during rollover, you will create a price gap, and vice versa. There is no way to avoid both, and so both cases involve look-ahead bias. Hence professional futures traders stick to actual contract prices.

You said : If your strategy generates trading signals based on theprice difference between two contracts, then you must choose the priceback-adjustment method; otherwise, the price difference may be wrong andgenerate a wrong trading signal..... However, if your strategy generates trading signals based on the ratio of prices between two contracts, then you must choose the returnback-adjustment method"

But, after reading what you replied above, saying that professional futures traders stick to actual contract to backtest, and manage the roll over to the next contract by buying/selling on the rollover day.

I'm a little bit confused of what should be the correct method to backtest, Let say I have 3 years data of future contract of commodity A and commodity B, and the data shows that Feb and August are the active contracts for both of A and B usually. Therefore I pick 2012 Feb, 2012 Aug, 2013 Feb, 2013 Aug, and 2014 Feb, 2014 Aug for commodity A , same for commodity B for backtesting

Based on what you mentioned above, I shouldn't do any back-adjustment of the return series from A and B before applying any ADF, johansen Test to it? Please kindly advise. Thank You

Hi Karen,If you have generated a vector of trading signals (buys or sells), you should use actual contract prices to determine the returns based on these trading signals.

But in generating a trading signal at time t, you can back-adjust futures for all time t' <= t to create a continuous contract. In particular, if you want to run a Johansen test on spreads of different futures, the backadjustment should involve only additions. If your signal is generated based on ratio of prices, the backadjustment should involve only multiplications.

Ernie, thanks for the blog and information. About the 'CEF Premium Reversion' strategy I wonder if you have an explanation why the different price changes the outcome significantly? I understand that the price is different but why does it result in a disadvantage, in particular when you trade long and short positions? Thanks, Niels

Ernie, I see your point on figuring out the correct open and closing prices in order to conduct a proper backtest. This is where I wanna make a suggestion of forgetting ohlc data, I find liquidity way more important of an issue unless you only trade 100 share or 1 contract positions. Volume based bars or tick bars which are constructed off 1min may be a little better.

As an example you would trade 500 share bars, and if you need to execute at open you would take the worst price of the range or median price of 1000share bar range as your price of execution.(latter is better because in reality even if you tried to execute this 500 share trade you wouldn't have been able to within first bars range because that would assume you were only buyer).My point is all these back tests are pointless without tick data - period.

Hi Slav,Yes, your idea for using a volume bar at the open is certainly a good alternative of using the official open price, which often has limited size. However, even though that auction size is limited, it isn't quite limited to 100 shares. You can typically trade at least $200K to $500K order depending on the market cap of the stock without moving the auction price.Ernie

Hi, I'm following your book codes (strategy "Buy on Gap" pg.93-96) on Matlab but as I'm using the 2016 version I had to rearrange some of your old functions (i.e "lagmatrix" instead of "lag" and "tsmovavg" + "movstd" instead of your personal functions for moving averages and moving standard deviations").Anyway...I honestly find very different results from yours even if using the very same data!Another thing I noticed is it seems you're not following your own conditions: for example:1) You calculate 90 days moving average for daily returns (close to close) and say that we have to consider only (today open/yesterday low) returns < than the 1 standard deviation from 90 days movaverages just computed. But...in your code you make the previous day low - 1 standard deviation. Isn't it wrong??

2)in the second condition you say that today low must be < than 20 days mov.average of closing prices. So, why in the code I read ma= moving average of close to close returns????

Please, I hope you can help me as I'm just a newbie matlab user and I really love to follow your work. Thanks

Thanks for your reply!Yes you're right about the point 2): the moving average is computed over closing prices and not return (sorry yesterday I was clearly done with my mind!!).The point "1" instead...I'm still not sure about it: Shouldn't it be: retgap < stdretC2C90d? Anyway, I'll work on it.

The real problem is (for the previous and future strategies I'll read on the book) working with different versions of Matlab (as I can't use any longer function as "lag" etc.) but I thought I managed to change functions in a way to get close to your results. I was clearly wrong (my Sharpe Ratio is only 0,63 compared to your 1,5 and my APR 0,0048 instead of 8,7%!!!). I don't know where, but I've failed into coding. So, is there by any chance, an update version of this book's Matlab code?Thank again for your attention and your patience!Beppe

Believe it or not, I made the same changes before your reply!!!Well...to be honest, "almost" the same: I forgot the "minus" sign. Ironically I got a far better APR (more than 20%) while Sharpe almost equal. Unfortunately I still have values too far from yours. It's clear that depends on functions and by some changes I made. No problem, I think for the moment it's more important understand the "meaning" of a strategy. So now I can go on with the book.

Now my very last question: "retO2C=(cl-op)./op; pnl=sum(positionTable.*(retO2C), 2); ret=pnl/topN;"You sum returns for every day in order to have a final column vector pnl. It's all ok but...summing percentages from different prices shouldn't require us to multiply these percentages for the opening prices? I mean, that way we have the absolute value we're really gaining/loosing. Or maybe I'm doing confusion because I don't even understand why dividing for 10.Anyway, I abused of your patience so thanks again for your help.

Beppe,Yes, my total returns are an arithmetic sum, not a compound return. This is useful for Sharpe ratio calculation, or if the trader wants to bet a fixed amount of capital each day.

However, even if you were to compute compound return, it isn't correct to multiply the price with the returns. They are totally different animals. Compound return formula would be just cumret=cumprod(1+retO2C)-1.