Thursday, April 04, 2013

I have come across many software platforms that allow traders to first specify and backtest a strategy and then, with the push of a button, turn the backtest strategy into a live trading program that can automatically submit orders to their favorite broker. (See all my articles on this topic here.) I called these platforms "Integrated Development Environment" (IDE) in my new book, and they range from the familiar and retail-oriented (e.g. MetaTrader, NinjaTrader, TradeStation), to the professional but skills-demanding (e.g. ActiveQuant, Marketcetera, TradeLink), and finally to the comprehensive and industrial-strength (e.g. Deltix, Progress Apama, QuantHouse, RTD Tango). Some of these require no programming skills at all, allowing you to construct strategies by dragging-and-dropping, others use some simple scripting languages like Python, and yet others demand full-blown programming abilities in Java, C#, or C++. But which of these allow us to backtest and execute high frequency strategies?

To state the obvious: backtesting HFstrategies is quite hard. The volume of data is one issue. But in addition, the execution details are very important to such strategies: details such as the exact exchange/venue to which we are routing our orders, the precise state of the order book that triggers our orders, the order types we are using, and finally the probability of getting filled if we use non-marketable orders. Messing up one of these details and the backtest will be far from realistic. I often tell people that it is easier to paper trade a HF strategy than to backtest one. While many of the platforms I reported above do allow backtesting using tick data, I don't know that they enable backtesting using the full order book and choice of execution venue. With this background, I am happy to report I have recently come across just such a platform called Lime Strategy Studio.

First, the bad news. LimeTrader is useful only to traders who trade with Lime Brokerage, as it is configured to send live orders to Lime only. [UPDATE: I have since learned that there are adapters available for 3rd party brokers.] However, if you are going to trade HF stocks and futures strategies, why not go with Lime, since they provide you with a comprehensive API, direct ultra-low latency feeds from the exchanges, and allow (nay, insist on) colocation either at the exchanges or at their data center at a reasonable fee? (Full Disclosure: I have no current business relationship with Lime, though I was a customer.) Another piece of bad news: the specification of the strategy must be in C++.

But once you get over these two hurdles, the benefits are manifold. Every detail that you can specify for a live trading strategy can be specified for the backtest and paper trading. As I said, these details may include order type, trading venue, state of order book, and even statistics of the order book, not to mention fundamental data such as earnings, corporate actions, and other user-provided data such as news. A fill simulator is included for your non-marketable orders. As with other IDEs, once you backtested a strategy in its every detail and are satisfied with the performance metrics, you can go live (either for paper or production trading) with the push of a button.

If any reader know of other IDEs that have similar features and useful for backtesting HF strategies, please let us know!

===

Speaking of HF strategies, traders often lament the ultra-high secrecy around them and the difficulty of gathering knowledge in this field. A friend (hat tip: Dave) referred me to this paper by Prof. Dragos Bozdog et. al. that gives a flavor of what sort of modeling may be involved. I find it very readable and thought-provoking.

78 comments:

Anonymous
said...

Could you share your opinion on PRIMUS Trade? Maybe recommend a similar but better (or more cost efficient platform)? It is similar to the platforms listed above and the company I work for currently uses it. Link to the platform: http://www.primustrade.com/

Heya Ernie, thanks again for an informative post. Is there middle ground for developing a HF trading system from a standing start, other than opening an account with Lime? ie -- is there a (recommended) source for tick data that can be downloaded and backtested against offline, and how does bootstrap an analytics platform/IDE with it?

Anon,I took a quick look at PRIMUS, but am not sure that it is suitable for HFT. I do not see any mention of access to historical order book info, nor the ability to direct an order to specific venue only.Ernie

experquisite,If you are patient enough, it is possible to backtest HF strategies using Matlab/Python, but you still won't have a good fill simulator. Of course, this won't matter if you are sending MKT orders.

If your strategy involves the order book, this would become even more difficult.Ernie

Understood, but where is the best place to get the raw data if you don't want to choose your broker yet? I suppose the most effective thing would be to get the tick data archives into an amazon instance, but then you have to choose the data format, the provider, all these details -- is there any primer on this sort of thing? Should one be targeting some kind of KDB, or HDF5, or something else?

For prototyping without orderbook (assuming fill simulation is not the most important aspect right now), can one still use python pandas or does it hit a wall? What about strategies and transforms to operate on live 'stateless' data, aka using exponential moving averages instead of simple, except for other common stats operations?

Hi Ernie, I am from Hong Kong and nice to meet you:) Why do you focus on MATLAB but not building a total new system with C++/C#/Java? Isn't that MATLAB focuses on complicated math calculation for research kind instead of real time trading?

Hi Anon,Indeed it is much more efficient to do research and backtesting in MATLAB than in C++. For low frequency executions MATLAB also works well. We only need C++ if we want to build a HF executions systems.Ernie

Hi Ernie, one IDE worth taking a look at is Multicharts (http://multicharts.com). They recently released a .NET version where you can design indicators and signals in C#. I just downloaded a free starter version (just for 2 symbols) and I'm trying it right now with Interactive Brokers. So far so good.

As a discretionary retail trading platform, it is not horrible. For algorithmic trading, you really don't need it if you are already using Matlab or Python.

Something to keep in mind about these retail platforms is that they are not optimized for performance, they are just optimized to work with high latency broker apis like IB's. Plus, neither multicharts nor tradelink have fix connectors. So, you can't plugin your broker via fix unlike with matlab or python. Still, I would say that metatrader is at least, semi-decent, compared to the rest.

If you really need faster execution, outside of paying for some all-in-one suite from quantfactory, tbricks or deltix, it's best to just go the EMS route. From what I heard about all-in-ones, quantfactory and tbricks tend to be cheaper than Deltix (you can look into that). So, you can still model strategies in matlab, but for execution, you can use something like xtrader, cqg, bloomberg emsx, etc.

Also, here is something else that you might want to look into. It is a demo of Simulink being used for financial modeling:

How do you have a high frequency system at home? I think:1. A normal home internet speed may not have HFT kind of speed2. Home computer may not be fast enough to do that kind of real time analysis/offline backtest.3. High frequency system would take $0.001 profit in each trade, while retail brokerage would charge much higher than that for each trade.

Welcome to share how can you build that, I thought it is only possible with big financial institution.

Hi HK,You can't of course literally run HFT strategies "at home". But you can build such strategies and run them at a collocated server.

It is also true that most retail brokerages would not be suitable due to high commissions or high latency. For stocks, you need a larger account so that you can trade at Lime or some other institutional brokerage. For FX and futures, there are more retail brokerages that allow you to collocate and have reasonably small commissions.

When I said HFT, I am referring to trades that take place between 1-100 ms. For sub-ms trades, I don't think retail traders should attempt them.

Lightspeed is a retail broker, it also offers HFT capabilities and co-location.

The Tradelink project overadvertises its capabilities. It doesn't have a working fix connector. What they have is a 'blank template', basically you need to make a fix connector. Very deceptive. Plus, it is a very unstable program.

Have you considered trying out Simulink/Stateflow for order book and fill simulation?

It is perhaps possible to build a realistic fill simulator using Mathworks Stateflow, but I would rather leave that to 3rd party firm or a broker to do. Especially for Forex, each broker/ECN will need to be tuned independently for this simulator to work.

Thanks for your answer. I am reading your book and you mentioned a good profitable trading strategy can be/should be "Make everything as simple as possible, but not simpler". Is that because a very complicate strategy would be curve fitting?

Why do banks/ibanks hire proprietary traders to manually trade, especially we have much higher technology for trading in these decades? I saw a main casino company in Macau hires proprietary trader trainee to trade stocks/derivative in HK market. The post mentioned they taught systemic method, unfortunately I couldn't get an interview for this job.

Here is a little history of me:I had interviews for trader assistant of a worldwide bank and trading system developer of a main global trading system company, both interviews were many rounds and hours but finally I couldn't get it. My IT career is alright and I keep moving on but never had a chance to be in a serious trader related position. I was in an OTC derivative broker trainee position but it was all about trading speed and relationship, that's not what I am good at and want to be so I got another job.

A discretionary trader may beat an algo system if 1) the trader possess fundamental market knowledge that the algo system doesn't, or 2) the market characteristics change rapidly so that the algo system cannot catch up but the infinitely adaptable human can.

But a lot of times, the "human skills" are illusory and may simply be good luck. The realization of this leads to the proliferation of algo trading across the globe.

I see, so only very few human in the world can be true master of manual discretionary trading under high pressure with super rational thinking and fundamental knowledge, and there is almost no way to tell if it is luck or real skill. Thanks and I am reading your book again:).

What is the better way to analyze quotes, in the raw form or consolidated into, say, 1 sec OHLC? If I do not consolidate then the number of quotes per given time interval will be different e.g. EUR/USD 2000 quotes 9 am-10 am but 1200 quotes 3 pm - 4 pm. Also, if I do not consolidate then the number of quotes will be different for different symbols, so how can I calculate their correlation?

Hi HK,Actually, I am not sure what you are going to use Monte Carlo methods for?Generally, I don't recommend using any fancy methods for quant trading. The simpler the method, the likelier it will actually make money.Ernie

HK,Sorry, I don't agree with most of what Seykota, Elliott, Gann, and DeMark said. My trading principles are mostly based on academic research coming out of the finance departments of various universities and the research desks of the US Federal Reserve Banks.Ernie

In theory, we might say that the markets are abnormal in the aftermath of a surprise, and we shouldn't trade our models. In practice, it is hard to determine what exactly constitutes a surprise, and I have seldom found any benefits in overriding one's models.Ernie

So basically we don't need to care financial news at all? Would you focus on corporates' P/E, dividends, and government release data? For me, I think both corporates and government can release not real data or hide whatever they want, especially government has the authority to make stuff up to stimulate the market or control crowd expectation. It would be hard to fake dividends in long term because corporates really need to pay the money.

Hi HK,Your question was about surprises, not about financial news.In the aftermath of true surprises, there is not much we can do. But for expected news releases, we can refrain from trading ahead of time, and this can be part of the trading strategy that can be backtested.

I get your meaning. So the effect of expected news like FOMC meetings can be back-tested and be part of a strategy. It doesn't matter if gov releases real data reflects the true economics or not, the effect can be back-tested.

Comparing to stock market, would FX future market be more predictable and has trend characteristic?

Stocks in Hong Kong are very tricky and not fair at all. The stock exchange over here is very loose. I trade ETF and futures but don't like to trade stocks in Hong Kong, because a stock of a single company is so risky even not small stock. Stocks that constructs Hang Seng index can be very risky too, like 330 ESPRIT or 267 CITIC PACIFIC.

If a strategy with no more than 5 variables is the best, then why financial institutions keep spending so much money to hire phd quants, programmers, traders for high/low frequency trading? Those financial giants should have knew what could be the best.

Hi HK,Most of the quants in investment banks are hired to model derivatives and fixed income models, which are highly nonlinear, and much more complex than cash equities or spot FX models. For the latter 2, linear models are preferred.Ernie

It is so hard to even find a job post for getting into proprietary trader or related programmer industry. almost all are always hiring very experienced people.

Look like banking are hiring less people for years after financial crisis. It has been many years but look like it is still so bad. I am not sure why especially I am in Asia and economics over here doesn't look like that bad.

Just read the news from Dow Jones, EBS is going to deploy a new system to rein in high-frequency traders...see below--------------------------------Foreign-exchange platform EBS is considering strict new handicaps on high-frequency trading.ICAP PLC (IAP.LN, IAPLY)-owned EBS is planning changes to how it processes orders to buy and sell currencies that it says would eliminate certain advantages for the fastest traders.The platform would do away with the "first in, first out" system of order processing, which gives the trader who places an order first access to the most favorable exchange rate. EBS says this system has been exploited by increasingly nimble programs that can buy and sell positions in microseconds, before other market players can react. Instead, EBS would bunch client orders together as they come in and execute them randomly, a spokesman said.EBS is considering launching the new system by summer, the platform confirmed.High-frequency trading has become an increasingly significant part of the foreign-exchange market in recent years, and it now makes up about 40% of all trading, nearly double the share in 2007, according to consultancy Aite Group. These firms have been able to pump out orders and snap up prices at such speed that they have nibbled away at banks' large orders and made it difficult for them to complete large trades at the price they want.EBS has taken steps to rein in high-frequency traders before, as its core clients, including large banks, complained that the platform wasn't doing enough to protect their ability to trade. Last September, EBS said it would expel users who launch a swarm of rapid-fire trade requests they have no intention of completing. The tactic is a common one for high-frequency algorithms, which use it to gauge the immediate direction of a market, while frustrating lower-speed traders, as it pre-empts their efforts to execute sizable orders. The exchange also dropped the fifth decimal place in its currency quotes, removing most of the finer price points that work best for high-speed computer-based traders.Spot foreign-exchange volume on EBS was down 2% in March from the previous year and 23% from 2011. Meanwhile, currencies-trading volumes on systems owned by EBS's top competitor Thomson Reuters Corp. (TRI, TRI.T) have grown since Thomson Reuters bought FXall, another dealing system.On April 18, Tradition, a brokerage, launched a rival currency-trading system called ParFX. Tradition's platform is backed by some of the biggest dealers in the $4 trillion-a-day foreign-exchange market, including Deutsche Bank AG (DB, DBK.XE), Barclays PLC (BCS, BARC.LN) and UBS AG (UBS, UBSN.VX).

Hi,I think that discussion should begin by explaining what HFT is. Even if you're using tick data, it still may not have to much with HF. There is a lot of misunderstandings, but of HF strategies typically require management of queue in order book.

R, Matlab and python are commonly used in HFT. It is important to be able to mix script code with C/C++. To test you need exchange simulator (Tradestation, Amibroker etc. completely unsuitable for this).

I am in to FX Spot algo trading since 5 years and I am heading towards HFT and MFT more and more.

I don’t know if you have time to answer this, my problem is that I don’t know of wich platform to work with.

Problem is that I am working with mql4 algos and I have a bunch of HFT-MFT that theoreticly and in backtests work very well.

Research gives that mt4 brokers, no matter if they claim to be ecn. Stp, mtf, exchange and/or if they are connected to currenex, still seems to filter tick data live feed.

Comparing tick data from mt4 ecn brokers including LMAX wich claim to be MTF and the first exchange broker on spot fx retail, I still see that tickdata differs a lot when compared to LMAX API .net plattform. Analyse shows that there are 10 times more ticks/sec on LMAX api .net solution

I need to have a backtesting ability, a paper trade ability and also ability to optimize the algo. And it has to be a platform that supports Spot FX.

Do you know wich plattform that supports all this?

And also: I have noticed that you have a course in MFT trading in October, would that one be suitable for me?

I am very interested to participate, but I assume it’s a question of platform and language to work with.

IF source code size is very big,CPU needs time to decode it and Quote chart will become very slow, How to slove this issue? ( I have not supercomputer)

"On his last day of work at New York-based Goldman Sachs in June 2009, Aleynikov uploaded hundreds of thousands of lines of source code from the firm’s high-frequency trading system"http://www.bloomberg.com/news/2012-02-17/ex-goldman-programmer-s-conviction-overturned-on-appeal.html

Do you still think Lime Strategy Studio is the best HF trading platform? I hear there are a lot hassles to working with the Lime brokerage people. Is there another tool I should look at? C++/C# are fine but I'd like to avoid Java.

I have not used Lime Strategy Studio for much work myself, and no longer trades through Lime Brokerage since our focus has turned to FX. However, I had found that their traders were very supportive of us.Ernie