The Sorcerer’s Apprentice, but with money: another look at HFT

A new series of visualizations from a market data feed firm paints a true …

"Why don't you know BATS [the third largest equity market in the world]?," Bernard Donefer, a finance professor and HFT expert at CUNY's Baruch College, asked me rhetorically. "Because there's nothing to look at. It's based in Kansas; the computers are in Jersey City." -- The Matrix, but with money: the world of high-speed trading

In the opening part of my first article on the bizarre and fascinating world of high-frequency trading, I went to some lengths to debunk the notion that "the stock market" looks anything like those still-popular press and movie images of gel-haired, frantic young traders screaming at each other across a large, open trading pit. Today's markets are essentially electronic networks, where humans and machines trade against each other via terminals and server racks.

But just because the markets are electronic doesn't mean you can't take a picture of them. Market analysis firm Nanex has produced some very compelling visualizations of the type of sub-second trading that goes on in our electronic exchanges. A number of the pictures are from BATS—a fully electronic exchange that consists of a bank of computers in Jersey City—but others cross multiple exchanges.

Nanex's visualizations page contains an ongoing catalog of the traces that stock-trading bots leave in the market, as they move individual symbols up and down in price thousands of times per second. The obvious, repetitive, algorithmically generated patterns produced will be familiar to any physicist or electrical engineer—the stock market is a now a networked collection of high-frequency oscillators and feedback loops of varying sizes, where computers cycle stock prices by the millisecond.

To get a sense of how this looks without clicking through to the Nanex page, take a look at the charts below for SHG, an NYSE-traded company company called the Shinhan Financial Group.

The Google Finance chart above shows the trading activity for July 14, 2010. Clearly, nothing unusual happened—it looks like a typical stock on a typical day.

But over the course of one second, at exactly 9:36:26 ET, a rapid-fire series of 760 quotes produced the following pattern:

Nanex calls this pattern "the Blue Bandsaw," and it's a small sample of a longer 11-second period in which 15,000 quotes were pushed through the exchange and the price oscillated in this cyclical pattern.

The pictures above may cause you to wonder if there's any meaningful relationship between the Google Finance chart and the Nanex chart. In other words, does the algorithm-generated activity that happens at the sub-second level affect the overall course of the stock on longer timeframes (a day, a month, a year), or are the two kinds of trading more or less independent?

Proponents of HFT argue that they are independent—that such millisecond fluctuations in a particular stock's price don't affect that same stock's price on a time scale that longer-term investors care about. Critics of HFT find it preposterous that such a fundamental change in the very structure of the markets has either a neutral or benign impact on long-term investors; in the wake of the May 6 "Flash Crash," those critics have gained some clout.

Informal conversations I've had with money managers and traders indicate that in the wake of the Flash Crash, even the insiders are scared of what the markets have become. Still, they have no choice but to keep trading—not only must they keep trading, but everyone is quietly stepping up their automated trading efforts to avoid getting eaten alive by their competitors' machines. It's a bit like The Sorcerer's Apprentice, except that there's no master magician who can step in and save us in the end.

Wherever you come down in the HFT debate, everyone with any money in the stock market would be well served to take a look at Nanex's pattern collection, so that they get the antiquated image of the trading pit out of their heads and get a real picture of the market that they've entrusted their assets to. Caveat emptor.

If you're interested in more details of the patterns and the backstory on how they were uncovered, Alexis Madrigal at The Atlantic has a great story up on them. Zero Hedge actually had this story a few days before Alexis, but their chart dump doesn't have nearly the amount of context.

It's all about being so fast that you can be the middle man even when the buyers and sellers show up at practically the same time. There's no way that adding middle men can improve efficiency.

I'm not sure I buy the arguements for HFT, but middle men serve a purpose. Is it more efficient for food manufacturers to sell to 100 million households, or for them to sell to 30 supermarket chains and those chains to sell to households?

The most common claim in favor of HFT is liquidity, but even if its true I'm not sure the cost is justified.

Nice article Jon. Very non-alarmist and to the point. Market analysis is pretty interesting, trying to figure out what's going on.

Quote:

Other than Congress. A 100% capital gains tax rate on positions held less than, oh, 30 seconds, ought to put this to rest.

Yes, tax it more! We already exported all the jobs in this country, lets export whats left of the financial industry too! LOL If the trade made money its already being taxed, so really, what does this gain?

Just outlaw computers and go back to the specialist floor system if you don't like it. Ooops, that model can't possibly work for the volumes that are occurring. That, combined with the fact that Uncle Sam has his hand in everyones pocket, and lower profit margins mean lower tax income. NY is already charging smokers $10 / pack for cigarettes, I don't know if they could get much more, so they would have to make up that tax revenue loss somewhere. I've heard taxing air might be possible!

Sorry to be so condescending, but taxes aren't the answer here. There's no real concensus that there is even a real problem here other than emotions. The issue is a little like Nascar. The bigger, well financed teams will usually place higher and are more consistent in winning than the smaller teams. Over the long haul, it makes a big difference. The average trader would never see the sub-penny fluctuations.

The circuit breaker rule will take care of the diving off the cliff scenario.

"We already exported all the jobs in this country, lets export whats left of the financial industry too!"

There is simply no valid reason for society, especially in light of flash cresh events, that HFT needs to be happening in our capital markets. The capital markets are designed to put those with capital to risk in touch with those that seek that capital. Thats it.

"Ooops, that model can't possibly work for the volumes that are occurring."

The volumes are occurring BECAUSE of the HFT.

"That, combined with the fact that Uncle Sam has his hand in everyones pocket, and lower profit margins mean lower tax income."

Trivially lower. What we gain is confidence that the system isn't rigged and people once again trust the market to function accordingly.

"There's no real concensus that there is even a real problem here other than emotions. "Yeah, May 6 didn't happen at all.

"It's a bit like The Sorcerer's Apprentice, except that there's no master magician who can step in and save us"

Other than Congress. A 100% capital gains tax rate on positions held less than, oh, 30 seconds, ought to put this to rest.

Yes, that's probably not a realistic solution, but on the surface it would lead to a reduction in HFT. However, I can just imagine how all these trading companies would just find ways to skirt around the issue. One possibility I just came up on the top of my head is to create 2 sister companies and trade amongst themselves and somehow hide/share/cancel out the profit temporarily. Never underestimate the creative solutions that people come up with when money is up for grabs :-)

I love to see this subject given attention. It is a very complex issue, with very little visibility to the public, and naturally, the knee-jerk reaction is to destroy it.

I can think of no less than 3 reasons to dislike the whole thing, My rational mind questions the best of human talent being incentivized towards the banking industry rather than pursuing more practical careers in science and engineering. There's the emotional "People get to make how many microtransactions per second with what kind of advantage I can't have?" And then there is the socialistic "the kind of control and accumulation of power and wealth on display here is fundamentally indefensible."

Without articles like this building paradigms for us, we cannot even begin to rationally think on the subject, much less fairly consider how to handle the issue. I personally would like to see the infrastructure used to implement these vast financial battlebots, if that information is out there.

There is only one real reason I can see why HFT might be dangerous to the markets as a whole; feedback between competing systems causing massive volatility. However, the majority of HFT systems are built around the whole principle of stopping this (ie, that trends tend to be mean reverting; so if a stock falls abnormally low, buy it immediately), so there would either need to be something extremely questionable going on, or a plain error (which is the excuse I've heard for the May 6th loveliness).

As for is it right that 'he who has the fastest computer/algorithm makes the money', or 'we should tax all transactions under 30 seconds'; what would that really solve? There is a certain amount of money available for HFT traders to exploit in the markets; namely the small edges which large flow/retail investors can't be bothered to chase up, such as the 1c between the bid and offer price on an option. They don't care, but a HFT firm who makes the spread 10,000,000 times a day might.

If you did somehow manage to ban HFT, what would happen? The banks would just take the cut instead, and it would be a much bigger cut as well.

As for the infrastructure used, the information is out there, but imagination will probably suffice as well. HFT is worth billions, and he who is fastest makes the most.

To call HFT's a supermarket for farmers is disingenuous. A million transactions are happening per second, many cancelled and many followed through with. An electronic "middleman" that buys and sells so you can't see who you are selling to. They make money on "see this person is selling and this person is buying, buy from X and sell to Y, cancel if not for prices. These things are done at milli seconds, there is no liquidity gain - the buyers and sellers would have found themselves without this intervention. The HFT's don't buy alot of stocks and sell them later giving liquidity, they leech incremental money off of buyers and sellers by having the fastest connection to the "floor". Often the market board itself sells optimum spots for premium rates, its a bit twisted.

If you want to go old school economics. Ask yourself "What service do they provide to make the money they do?" Or are they using an bought advantage that everyone cannot have?

As a side note based on experience I would say that many of the weird moves on those charts can be attributed to incompetence, not malice. Pattern detection is tricky to do fast and some people don't get it right or don't add such a thing due to poor design choices.

I remember when exchanges would spot such things and threaten people with temporary bans, it would seem like most have stopped, perhaps due to the high volumes.

It's all about being so fast that you can be the middle man even when the buyers and sellers show up at practically the same time. There's no way that adding middle men can improve efficiency.

I'm not sure I buy the arguements for HFT, but middle men serve a purpose. Is it more efficient for food manufacturers to sell to 100 million households, or for them to sell to 30 supermarket chains and those chains to sell to households?

The most common claim in favor of HFT is liquidity, but even if its true I'm not sure the cost is justified.

In this case though the "middleman" follows you into the grocery store, sees you eyeing the last tube of toothpaste on the shelf, grabs it, buys it first, then offers to sell it to you at a mark up.

Really, I think parasitism is a fair and accurate description of what goes on.

It's no secret that regulation trails innovation. Eventually market regulators will find ways minimizing the negative effects of HFT and retaining the good things about it. Then trading firms will find ways around the controls, and then the controls will be modified, then ... wash, rinse, repeat.

This was the case with face to face floor trading and will be the case with electronic trading.

So I have not seen one cited "evil" of HFT listed here yet. The closest we are getting is some artificial volatility and the May 6th incident who's exact cause it still unknown. It could be entirely possible that the remaining humans in the system make mistakes or highly unusual orders and all the computerized traders sweep in.

About the toothpaste example, that's a little contrived. More likely some large investor sees an undervalued stock and starts to buy a large quantity. The HFT systems detect his buying pattern and step up orders for the stock to make some money off his buying. In doing so they bid up the price of the stock, which is exactly what should be happening because if such a large investor think the stock is undervalued it must be under priced.

So I have not seen one cited "evil" of HFT listed here yet. The closest we are getting is some artificial volatility and the May 6th incident who's exact cause it still unknown.

That it was largely driven by gain>1 feedback among HFT, for at least 700 points of the 1000 point drop, is commonly understood. Automated triggered sells based on price likely contributed. Exactly how it happened is shrouded in propriety trade secrets distributed around the HFT [, and in private account settings of millions of people and entities,] which is actually part of the issue (AKA "evil").

The article is a bit misleading. The plot shown is from actual data, but the assertion that "the price [of the security] oscillated in this cyclical pattern" is incorrect. While the text is difficult to read, note that the blue trace with the strange pattern is labeled as "NSDQ Bid." This trace refers to the bid price posted by the NSDQ market maker over the time period in question. However, the green trace, "BBid Price," refers to the bid component of the National Best Bid and Offer (NBBO), which is what you're supposed to get if you were to execute a market sell order during that time period. In this case, the NBBO was not affected by the strange pattern of bids.

That's not to say that there aren't other examples where the NBBO was affected and therefore the algorithms on display actually had a tangible (although transient) effect on the market. I don't dispute that the behavior noted in the article is odd and the notion that it adds anything to the market is questionable. And, as is pointed out, a crowd of similarly-minded trading bots, who make up the majority of trading volume on exchanges, can have unpredictable interactions that could lead to significant events. I just want to point out that the effects of this particular example alone are negligible; by itself, it was not affecting the market quote.

I believe that calling these events a result of "a networked collection of high-frequency oscillators and feedback loops", and relating them to feedback systems "familiar to any physicist or electrical engineer", is probably an overstatement.

Looking at Nanex's charts, only one (two, at most) are obviously the result of high-frequency feedback between different HFT agents. The overall quote stuffing sure is a skivvy strategy to gain a small advantage on regular traders, but you can't tell for sure it increases volatility. As for the "saw" and "knife" patterns, they appear to be the result of in-house isolated algorithms and I expect that if they are not well controlled for internal stability, it will be for that house's loss.

There are only 2 major issues with HFT I can see: 1) The trading is now pure gambling, completely disassociated from any company valuation; and 2) The trading occurs at Faster-Than-Real-Time speed, so that when the inevitable human errors and software glitches come up, hundreds of thousands of shares have already been traded and every investing company's algorithm has automatically responded to this accidental, fictitious data marker.

So my question is, why allow gambling when it's tied into our market index for shareholder prices? They could just as easily set up a legitimate gambling market that copies the stock market structure, where the prices are influenced by real-world stock negotiations but whose own prices do not reciprocate into the real world. It would be like playing an MMORPG based in a Google Earth virtual world - reality helps direct it, but blowing up a virtual mountain has no real-world consequences.

The article is a bit misleading. The plot shown is from actual data, but the assertion that "the price [of the security] oscillated in this cyclical pattern" is incorrect. While the text is difficult to read, note that the blue trace with the strange pattern is labeled as "NSDQ Bid." This trace refers to the bid price posted by the NSDQ market maker over the time period in question. However, the green trace, "BBid Price," refers to the bid component of the National Best Bid and Offer (NBBO), which is what you're supposed to get if you were to execute a market sell order during that time period. In this case, the NBBO was not affected by the strange pattern of bids....I just want to point out that the effects of this particular example alone are negligible; by itself, it was not affecting the market quote.

I think the real point is that such activity has a chance of slightly disrupting the trading mechanics of one particular exchange (as detailed in Nanex's explanation of quote stuffing), which may allow a trader who is ready for it to exploit millisecond-class discrepancies between the exchange bid/ask prices and the NBBO. If that is true, then it's unquestionably nefarious.

Another possibility is that these patterns are deliberate noise introduced to fool competitors, which is slightly less nefarious, but still represents an abuse. Finally, they may merely represent aberrant oscillations in the algorithmic dynamical systems that run HFT. But none of these cases represents desirable or useful market operation, so there's a strong case for regulators to track these down and stamp them out.

[Edit]BTW, excellent reporting Jon, would have totally missed this but for Ars.

Another possibility is that these patterns are deliberate noise introduced to fool competitors, which is slightly less nefarious, but still represents an abuse. Finally, they may merely represent aberrant oscillations in the algorithmic dynamical systems that run HFT. But none of these cases represents desirable or useful market operation, so there's a strong case for regulators to track these down and stamp them out.

I was under the impression that it was this. Deliberate noise. Since the offers for trades were way way out of line of anything acceptable these trades were offered up as noise to confuse competitors algorithms on machines that didnt have ping times fast enough to keep up; or tie up the cpu time of the trading system dealing with analyzing and rejecting these bids in order to keep competitors orders from going through on time

The real problem, IMO, is not HFT itself, but the general attitude which drives things like HFT. It's the attitude that you can get something for nothing in a very short amount of time. It's the same kind of thinking which keeps casinos in business. The only difference is that the people playing this game are doing it at lightning speeds with incredible amounts of money, and when they screw up, it affects us all.

How about we actually make laws to revert the stock market back to serve its original purpose? You know, sensible, long term investments made by people actually interested and knowledgeable about companies instead of high frequency speculation by people just looking to make a quick buck at everyone's expense. Imagine, for example, if you had to hold on to purchased stock for at least three months.

I've been under the impression that HFT makes money while a market is moving, mostly by jumping in between a seller and a buyer and siphoning off the difference in the price that occurred for the few fractions of a second between when the seller put the stuff up for sale and when the bidder put down his bid price.

The idea that the whole system is set up to encourage (and create!) price fluctuations so they can make a profit. The downside is that it makes the market fragile. The effects of a small error or honest devaluation become almost instantly magnified and spread across the market as a whole--which is good for HFT, since they make a whole bunch of money on the volatility.

They're not interested in investing in companies to help them grow, just in gaming the system as much as possible to siphon off cash wherever they can find it. This is bad because they don't care if the market crashes either, in fact it would be good, volatility means more profits. Crashing the markets every couple of years would probably be the best situation for them in fact. The only thing stopping them is the threat that the government might finally step in an start doing their job (regulating) if they make too much of a mess.

It's interesting to point out that a lot of the market swings we've seen have been due to HFT's.

In the past we would think "oh, everyone saw the news, paniced and sold." But these days, the HFT's cause all this cascade effect buying / selling. When Greece was going back-n-forth with their debt issue, you could see things like the S&P dip $1-2 in a couple of days, which is crazy and seems like there's nothing but alarmists on Wall Street.. But it's the threshholds set on the HFT's. One would hit a decision point, then cause a cascade effect amongst the others...and the "market" would plummet.

It's both fascinating and scary, since it's no longer rally based on skill, but who can buy the best programmers to make the best HFT AI, and that will just transact with "funny money" back and forth all day.

I don't think anyone has actually made a fair case for how HFT is any more of a "gamble" then normal investing. Just because you don't understand what metrics these companies use to detect value in a stock, doesn't mean it's gambling.

A computer that looks at patterns in demand and purchases for a stock is trying to measure an unmeasurable quantity, the stock price in the future, through a measurable quantity: the actions people make with the stock now (and probably a bunch of other inputs to). Computers do this on time scales from milliseconds to days, to weeks, to months. I fail to see how a computer model which tells me a stock is going to rise/fall in price in the next millisecond, is any different from larger scale models which predict a rise in a stock over the next few weeks or months.

I don't think anyone has actually made a fair case for how HFT is any more of a "gamble" then normal investing. Just because you don't understand what metrics these companies use to detect value in a stock, doesn't mean it's gambling.

A computer that looks at patterns in demand and purchases for a stock is trying to measure an unmeasurable quantity, the stock price in the future, through a measurable quantity: the actions people make with the stock now (and probably a bunch of other inputs to). Computers do this on time scales from milliseconds to days, to weeks, to months. I fail to see how a computer model which tells me a stock is going to rise/fall in price in the next millisecond, is any different from larger scale models which predict a rise in a stock over the next few weeks or months.

Proponents of HFT would agree with you. The concept behind trading so frequently is that you make many low-risk, low-reward trades. You don't make much on an individual trade, but because you make so many of them, you can make sizable profit. As others have pointed out, there are techniques predicated on the ability of some players to act faster than others, so that they can wedge themselves in the middle of a transaction and take a small cut. In addition, information availability isn't equal; so-called flash orders allow those who pay exchanges for the privilege to get a short advance notice of orders before they are executed. A market participant who could execute orders quickly enough could use this information to jump in and front-run buyers, pushing the ultimate price for the original buyer up.

I can't come up with a philosophical argument why HFT should be banned, but I also can't really understand why anyone would back it: it literally is using large sums of capital to leverage minor fluctuations in systems where there can be much larger impacts (i.e. the flash crash).

I think that the NYSE should simply put a short term hold on all trades (i.e. you can't sell what you bought for 2 minutes or 30 seconds or 5 minutes or something).

The "lower" end markets can then make their own decisions, in much the same way that pink sheets are traded: sure, you can't trade them on the NYSE, but you can trade them elsewhere.

Meaning this wouldn't be a tax rule, this wouldn't be an SEC rule, this would be a rule managed by the NYSE itself.

That way, if someone wants to participate in HFT they have somewhere to do it, but it won't affect the largest part of the system. The NYSE gets the stability they desire, the traders get the live-fast loose rules they desire.

Basicly the same thing as the flash crash happened when a broker got drunk an started buying a few million barels of oil last year. Trading is risk and most pro traders would be against any limits on how long a security position must be held.

Also your kidding yourself if you think the modivations four holding a position for months or years is any different than a few seconds or a fraction of one.

Basicly the same thing as the flash crash happened when a broker got drunk an started buying a few million barels of oil last year. Trading is risk and most pro traders would be against any limits on how long a security position must be held.

Also your kidding yourself if you think the modivations four holding a position for months or years is any different than a few seconds or a fraction of one.

I'm guessing you are replying to my comment given the nature of your post, but please quote the person you are responding to next time for clarity.

It goes without saying that any short-term trader is going to be against any kind of regulation that would prevent him from trading in the short-term. Why? Well, the answer to that gets to the crux of the issue. These short-term traders are parasites of our financial system who are out to make a quick buck at our peril. They don't serve the purpose of the stock market, which is to help companies gain capital from investors in exchange for a reasonable return in the form of dividends and/or increased stock value over the long term. Simply put, these parasites do not help this process. Instead, they simply game the system to make a quick buck, all while increasing the volatility of the market.

It does not matter whether it is a day trader making trades every fifteen minutes or a large investment firm with a lightning fast supercomputer making thousands of transactions per second; these people are parasites of the system. They do nothing to help the economy, nor do they help increase the capital of the companies being traded.

While the base motivations of real, long-term investors and short-term traders may be the same (to make money), the former actually helps our economy by giving companies needed capital. The latter are merely parasites who steal small amounts from the transactions of the former by acting as middle men. They are unnecessary gamblers who only serve to destabilize a necessary part of our economy.

Solution: one fix per week per stock. No more continuous quoting, no more day trading, no more crazy fluctuation on any rumor.This simplify things a lot, orders are queued all week long, and once a week, the stock value is fixed in order to maximize the number of transactions (like it was done ... few decades ago, I believe). The stock rise and fall depending on companies and sectors evolution, and real value is created.

Advantage: normal peoples are able to understand 'the market', and might trust it more with a significant part of their savings, allowing at least the middle-class to reap some benefits from the long term economic growth (because we all know the wages are stuck in neutral for the foreseeable eons).

Drawback (or is it?): 99% of the financial industry (the gabling part) will become irrelevant and thousands of Wall-Street and London-City cokeheads would have to find a new job. One where the actually contribute to society, not leach on it.

How about an exchange-enforced (i.e. legally mandated) and randomized 60-300 second delay in transaction processing. Unpredictable delays of 1-5 minutes should keep these systems from siphoning off actual worth while still being able to automate (and thus control and coordinate) the job a human would do.