Before the collapse of the U.S. financial system in 2008, Brad Katsuyama could tell himself that he bore no responsibility for that system. He worked for the Royal Bank of Canada, for a start. RBC might have been the fifth-biggest bank in North America, by some measures, but it was on nobody’s mental map of Wall Street. It was stable and relatively virtuous and soon to be known for having resisted the temptation to make bad subprime loans to Americans or peddle them to ignorant investors. But its management didn’t understand just what an afterthought the bank was — on the rare occasions American financiers thought about it at all. Katsuyama’s bosses sent him to New York from Toronto in 2002, when he was 23, as part of a “big push” for the bank to become a player on Wall Street. The sad truth was that hardly anyone noticed it. “The people in Canada are always saying, ‘We’re paying too much for people in the United States,’ ” Katsuyama says. “What they don’t realize is that the reason you have to pay them too much is that no one wants to work for RBC. RBC is a nobody.”

Before arriving there as part of the big push, Katsuyama had never laid eyes on Wall Street or New York City. It was his first immersive course in the American way of life, and he was instantly struck by how different it was from the Canadian version. “Everything was to excess,” he says. “I met more offensive people in a year than I had in my entire life. People lived beyond their means, and the way they did it was by going into debt. That’s what shocked me the most. Debt was a foreign concept in Canada. Debt was evil.”

For his first few years on Wall Street, Katsuyama traded U.S. energy stocks and then tech stocks. Eventually he was promoted to run one of RBC’s equity-trading groups, consisting of 20 or so traders. The RBC trading floor had a no-jerk rule (though the staff had a more colorful term for it): If someone came in the door looking for a job and sounding like a typical Wall Street jerk, he wouldn’t be hired, no matter how much money he said he could make the firm. There was even an expression used to describe the culture: “RBC nice.” Although Katsuyama found the expression embarrassingly Canadian, he, too, was RBC nice. The best way to manage people, he thought, was to persuade them that you were good for their careers. He further believed that the only way to get people to believe that you were good for their careers was actually to be good for their careers.

His troubles began at the end of 2006, after RBC paid $100 million for a U.S. electronic-trading firm called Carlin Financial. In what appeared to Katsuyama to be undue haste, his bosses back in Canada bought Carlin without knowing much about the company or even electronic trading. Now they would receive a crash course. Katsuyama found himself working side by side with a group of American traders who could not have been less suited to RBC’s culture. The first day after the merger, Katsuyama got a call from a worried female employee, who whispered, “There is a guy in here with suspenders walking around with a baseball bat in his hands.” That turned out to be Carlin’s chief executive, Jeremy Frommer, who was, whatever else he was, not RBC nice. Returning to his alma mater, the University at Albany, years later to speak about the secret of his success, Frommer told a group of business students: “It’s not just enough to fly in first class; I have to know my friends are flying in coach.”

Installed in Carlin’s offices, RBC’s people in New York were soon gathered to hear a state-of-the-financial-markets address given by Frommer. He stood in front of a flat-panel computer monitor that hung on his wall. “He gets up and says the markets are now all about speed,” Katsuyama says. “And then he says, ‘I’m going to show you how fast our system is.’ He had this guy next to him with a computer keyboard. He said to him, ‘Enter an order!’ And the guy hit Enter. And the order appeared on the screen so everyone could see it. And Frommer goes: ‘See! See how fast that was!!!’ ” All the guy did was type the name of a stock on a keyboard, and the name was displayed on the screen, the way a letter, once typed, appears on a computer screen. “Then he goes, ‘Do it again!’ And the guy hits the Enter button on the keyboard again. And everyone nods. It was 5 in the afternoon. The market wasn’t open; nothing was happening. But he was like, ‘Oh, my God, it’s happening in real time!’ ”

Katsuyama couldn’t believe it. He thought: The guy who just sold us our new electronic-trading platform either does not know that his display of technical virtuosity is absurd or, worse, he thinks we don’t know.

As it happened, at almost exactly the moment Carlin Financial entered Brad Katsuyama’s life, the U.S. stock market began to behave oddly. Before RBC acquired this supposed state-of-the-art electronic-trading firm, Katsuyama’s computers worked as he expected them to. Suddenly they didn’t. It used to be that when his trading screens showed 10,000 shares of Intel offered at $22 a share, it meant that he could buy 10,000 shares of Intel for $22 a share. He had only to push a button. By the spring of 2007, however, when he pushed the button to complete a trade, the offers would vanish. In his seven years as a trader, he had always been able to look at the screens on his desk and see the stock market. Now the market as it appeared on his screens was an illusion.

This made it impossible for Katsuyama to do his job properly. His main role as a trader was to play the middleman between investors who wanted to buy and sell big amounts of stock and the public markets, where the volumes were smaller. Say some investor wanted to sell a block of three million Intel shares, but the markets showed demand for only one million shares: Katsuyama would buy the entire block from the investor, sell off a million shares instantly and then work artfully over the next few hours to unload the other two million. If he didn’t know the actual demand in the markets, he couldn’t price the larger block. He had been supplying liquidity to the market; now whatever was happening on his screens was reducing his willingness to do that.

By June 2007 the problem had grown too big to ignore. At that point, he did what most people do when they don’t understand why their computers aren’t working the way they’re supposed to: He called tech support. Like tech-support personnel everywhere, their first assumption was that Katsuyama didn’t know what he was doing. " ‘User error’ was the thing they’d throw at you,” he says. “They just thought of us traders as a bunch of dumb jocks.”

Finally he complained so loudly that they sent the developers, the guys who came to RBC in the Carlin acquisition. “They told me it was because I was in New York and the markets were in New Jersey and my market data was slow,” Katsuyama says. “Then they said that it was all caused by the fact that there are thousands of people trading in the market. They’d say: ‘You aren’t the only one trying to do what you’re trying to do. There’s other events. There’s news.’ ”

If that was the case, he asked them, why did the market in any given stock dry up only when he was trying to trade in it? To make his point, he asked the developers to stand behind him and watch while he traded. “I’d say: ‘Watch closely. I am about to buy 100,000 shares of AMD. I am willing to pay $15 a share. There are currently 100,000 shares of AMD being offered at $15 a share — 10,000 on BATS, 35,000 on the New York Stock Exchange, 30,000 on Nasdaq and 25,000 on Direct Edge.’ You could see it all on the screens. We’d all sit there and stare at the screen, and I’d have my finger over the Enter button. I’d count out loud to five. . . .

“ ‘One. . . .

“ ‘Two. . . . See, nothing’s happened.

“ ‘Three. . . . Offers are still there at 15. . . .

“ ‘Four. . . . Still no movement. . . .

“ ‘Five.’ Then I’d hit the Enter button, and — boom! — all hell would break loose. The offerings would all disappear, and the stock would pop higher.”

At which point he turned to the developers behind him and said: “You see, I’m the event. I am the news.”

To that, they had no response. Katsuyama suspected the culprit was Carlin’s setup. “As the market problem got worse,” he says, “I started to just assume my real problem was with how bad their technology was.”

But as he talked to Wall Street investors, he came to realize that they were dealing with the same problem. He had a good friend who traded stocks at a big-time hedge fund in Stamford, Conn., called SAC Capital, which was famous (and soon to be infamous) for being one step ahead of the U.S. stock market. If anyone was going to know something about the market that Katsuyama didn’t know, he figured, it would be someone there. One spring morning, he took the train up to Stamford and spent the day watching his friend trade. Right away he saw that, even though his friend was using software supplied to him by Goldman Sachs and Morgan Stanley and the other big firms, he was experiencing exactly the same problem as RBC: He would hit a button to buy or sell a stock, and the market would move away from him. “When I see this guy trading, and he was getting screwed — I now see that it isn’t just me. My frustration is the market’s frustration. And I was like, ‘Whoa, this is serious.’ ”

People always assumed that because he was Asian, Brad Katsuyama must be a computer wizard. In reality, he couldn’t (or wouldn’t) even program his own DVR. What he had was an ability to distinguish between computer people who actually knew what they were talking about and those who didn’t. So he wasn’t exactly shocked when RBC finally gave up looking for someone to run its mess of an electronic-trading operation and asked him if he would take over and try to fix it. He shocked his friends and colleagues, however, when he agreed to do it, because A) he had a safe and cushy $1.5-million-a-year job running the human traders, and B) RBC had nothing to add to electronic trading. The market was cluttered; big investors had use for only so many trading algorithms sold by brokers; and Goldman Sachs and Morgan Stanley and Credit Suisse had long since overrun that space.

So Katsuyama was in charge of a business called electronic trading — with only Carlin’s inferior software to sell. What he had, instead, was a fast-growing pile of unanswered questions. Between the public stock exchanges and the dark pools — private exchanges created by banks and brokers that did not have to report in real time what trading activities took place within them — why were there now nearly 60 different places, most of them in New Jersey, where you could buy any listed stock? Why did one public exchange pay you to do something — sell shares, say — when another exchange charged you to do the same exact thing? Why was the market displayed on Wall Street trading screens an illusion?

He hired Rob Park, a gifted technologist, to explain to him what actually happened inside all these new Wall Street black boxes, and together they set out to assemble a team to investigate the U.S. stock market. Once he had a team in place, Katsuyama persuaded his superiors at RBC to conduct what amounted to a series of experiments. For the next several months, he and his people would trade stocks not to make money but to test theories. RBC agreed to let his team lose up to $10,000 a day to figure out why the market in any given stock vanished the moment RBC tried to trade in it. Katsuyama asked Park to come up with some theories.

They started with the public markets — 13 stock exchanges scattered over four different sites run by the New York Stock Exchange, Nasdaq, BATS and Direct Edge. Park’s first theory was that the exchanges weren’t simply bundling all the orders at a given price but arranging them in some kind of sequence. You and I might each submit an order to buy 1,000 shares of Intel at $30 a share, but you might somehow obtain the right to cancel your order if my order was filled. “We started getting the idea that people were canceling orders,” Park says. “That they were just phantom orders.”

Katsuyama tried sending orders to a single exchange, fairly certain that this would prove that some, or maybe even all, of the exchanges were allowing these phantom orders. But no: To his surprise, an order sent to a single exchange enabled him to buy everything on offer. The market as it appeared on his screens was, once again, the market. “I thought, [expletive], there goes that theory,” Katsuyama says. “And that’s our only theory.”

It made no sense: Why would the market on the screens be real if you sent your order to only one exchange but prove illusory when you sent your order to all the exchanges at once? The team began to send orders into various combinations of exchanges. First the New York Stock Exchange and Nasdaq. Then N.Y.S.E. and Nasdaq and BATS. Then N.Y.S.E., Nasdaq BX, Nasdaq and BATS. And so on. What came back was a further mystery. As they increased the number of exchanges, the percentage of the order that was filled decreased; the more places they tried to buy stock from, the less stock they were actually able to buy. “There was one exception,” Katsuyama says. “No matter how many exchanges we sent an order to, we always got 100 percent of what was offered on BATS.” Park had no explanation, he says. “I just thought, BATS is a great exchange!”

One morning, while taking a shower, Rob Park came up with another theory. He was picturing a bar chart he had seen that showed the time it took orders to travel from Brad Katsuyama’s trading desk in the World Financial Center to the various exchanges.

The increments of time involved were absurdly small: In theory, the fastest travel time, from Katsuyama’s desk in Manhattan to the BATS exchange in Weehawken, N.J., was about two milliseconds, and the slowest, from Katsuyama’s desk to the Nasdaq exchange in Carteret, N.J., was around four milliseconds. In practice, the times could vary much more than that, depending on network traffic, static and glitches in the equipment between any two points. It takes 100 milliseconds to blink quickly — it was hard to believe that a fraction of a blink of an eye could have any real market consequences. Allen Zhang, whom Katsuyama and Park viewed as their most talented programmer, wrote a program that built delays into the orders Katsuyama sent to exchanges that were faster to get to, so that they arrived at exactly the same time as they did at the exchanges that were slower to get to. “It was counterintuitive,” Park says, “because everyone was telling us it was all about faster. We had to go faster, and we were slowing it down.” One morning they sat down to test the program. Ordinarily when you hit the button to buy but failed to get the stock, the screens lit up red; when you got only some of the stock you were after, the screens lit up brown; and when you got everything you asked for, the screens lit up green.

The screens lit up green.

“It’s 2009,” Katsuyama says. “This had been happening to me for almost two years. There’s no way I’m the first guy to have figured this out. So what happened to everyone else?” The question seemed to answer itself: Anyone who understood the problem was making money off it.

Now he and RBC had a tool to sell to investors: The program Zhang wrote to build delays into the stock-exchange orders. The tool enabled traders like Katsuyama to do the job they were meant to do — take risk on behalf of the big investors who wanted to trade big chunks of stock. They could once again trust the market on their screens. The tool needed a name. The team stewed over this, until one day a trader stood up at his desk and hollered: “Dude, you should just call it Thor! The hammer!” Someone was assigned to figure out what Thor might be an acronym for, and some words were assembled, but no one remembered them. The tool was always just Thor. “I knew we were onto something when Thor became a verb,” Katsuyama says. “When I heard guys shouting, ‘Thor it!’ ”

The other way he knew they were on to something was from conversations he had with a few of the world’s biggest money managers. The first visit Katsuyama and Park made was to Mike Gitlin, who oversaw global trading for billions of dollars in assets for the money-management firm T. Rowe Price. The story they told didn’t come to Gitlin as a complete shock. “You could see that something had just changed,” Gitlin says. “You could see that when you were trading a stock, the market knew what you were going to do, and it was going to move against you.” But what Katsuyama described was a far more detailed picture of the market than Gitlin had ever considered — and in that market, all the incentives were screwy. The Wall Street brokerage firm that was deciding where to send T. Rowe Price’s buy and sell orders had a great deal of power over how and where those orders were submitted. Some exchanges paid brokerages for their orders; others charged for those orders. Did that influence where the broker decided to send an order, even when it didn’t sync with the interests of the investors the broker was supposed to represent? No one could say. Another wacky incentive was “payment for order flow.” As of 2010, every American brokerage and all the online brokers effectively auctioned their customers’ stock-market orders. The online broker TD Ameritrade, for example, was paid hundreds of millions of dollars each year to send its orders to a hedge fund called Citadel, which executed the orders on behalf of TD Ameritrade. Why was Citadel willing to pay so much to see the flow? No one could say with certainty what Citadel’s advantage was.

Katsuyama and his team did measure how much more cheaply they bought stock when they removed the ability of some other unknown trader to front-run them. For instance, they bought 10 million shares of Citigroup, then trading at roughly $4 per share, and saved $29,000 — or less than 0.1 percent of the total price. “That was the invisible tax,” Park says. It sounded small until you realized that the average daily volume in the U.S. stock market was $225 billion. The same tax rate applied to that sum came to nearly $160 million a day. “It was so insidious because you couldn’t see it,” Katsuyama says. “It happens on such a granular level that even if you tried to line it up and figure it out, you wouldn’t be able to do it. People are getting screwed because they can’t imagine a microsecond.”

Ronan Ryan didn’t look like a Wall Street trader. He had pale skin and narrow, stooped shoulders and the uneasy caution of a man who has survived one potato famine and is expecting another. He also lacked the Wall Street trader’s ability to seem more important and knowledgeable than he actually was. Yet from the time he first glimpsed a Wall Street trading floor, in his early 20s, Ryan badly wanted to work there. “It’s hard not to get enamored of being one of these Wall Street guys who people are scared of and make all this money,” he says.

Born and raised in Dublin, he moved to America in 1990, when he was 16. Six years later, his father was recalled to Ireland; Ronan stayed behind. He didn’t think of Ireland as a place anyone would ever go back to if given the choice, and he embraced his version of the American dream. After graduating from Fairfield University in 1996, he sent letters to all the Wall Street banks, but he received just one flicker of interest from what, even to his untrained eyes, was a vaguely criminal, pump-and-dump penny-stock brokerage firm.

Eventually he met another Irish guy who worked in the New York office of MCI Communications, the big telecom company. “He gave me a job strictly because I was Irish,” Ryan says.

He had always been handy, but he never actually studied anything practical. He knew next to nothing about technology. Now he started to learn all about it. “It’s pretty captivating, when you take the nerdiness out of it” and figure out how stuff works, he says. How a copper circuit conveyed information, compared with a glass fiber. How a switch made by Cisco compared with a switch made by Juniper. Which hardware companies made the fastest computer equipment and which buildings in which cities contained floors that could withstand the weight of that equipment (old manufacturing buildings were best). He also learned how information actually traveled from one place to another — not in a straight line run by a single telecom carrier, usually, but in a convoluted path run by several. “When you make a call to New York from Florida, you have no idea how many pieces of equipment you have to go through for that call to happen. You probably just think it’s like two cans and a piece of string. But it’s not.” A circuit that connected New York City to Florida would have Verizon on the New York end, AT&T on the Florida end and MCI in the middle; it would zigzag from population center to population center.

Ryan hadn’t been able to find a job on Wall Street, but by 2005 his clients were more likely than ever to be big Wall Street banks. He spent entire weeks inside Goldman Sachs and Lehman Brothers and Deutsche Bank, finding the best routes for their fiber and the best machines to execute their stock-market trades.

In 2005, he went to work for BT Radianz, a company that was born of 9/11, after the attacks on the World Trade Center knocked out big pieces of Wall Street’s communication system. The company promised to build a system less vulnerable to outside attack. Ryan’s job was to sell the financial world on the idea of subcontracting its information networks to Radianz. In particular, he was meant to sell the banks on “co-locating” their computers in Radianz’s data center in Nutley, N.J., to be closer, physically, to where the stock exchanges were located.

Not long after he started his job at Radianz, Ryan received an inquiry from a hedge fund based in Kansas City, Kan. The caller said he worked at a stock-market trading firm called Bountiful Trust and that he heard Ryan was an expert at moving financial data from one place to another. Bountiful Trust had a problem: In making trades between Kansas City and New York, it took too long to determine what happened to the firm’s orders — that is, what stocks had been bought and sold. They also noticed that, increasingly, when they placed their orders, the market was vanishing on them. “He says, ‘My latency time is 43 milliseconds,’ ” Ryan recalls. “And I said, ‘What the hell is a millisecond?’ ”

“Latency” was simply the time between the moment a signal was sent and when it was received. Several factors determined the latency of a trading system: the boxes, the logic and the lines. The boxes were the machinery the signals passed through on their way from Point A to Point B: the computer servers and signal amplifiers and switches. The logic was the software, the code instructions that operated the boxes. Ryan didn’t know much about software, except that more and more it seemed to be written by guys with thick Russian accents. The lines were the glass fiber-optic cables that carried the information from one box to another. The single-biggest determinant of speed was the length of the fiber, or the distance the signal needed to travel. Ryan didn’t know what a millisecond was, but he understood the problem with this Kansas City hedge fund: It was in Kansas City. Light in a vacuum travels at 186,000 miles per second or, put another way, 186 miles a millisecond. Light inside fiber bounces off the walls and travels at only about two-thirds of its theoretical speed. “Physics is physics — this is what the traders didn’t understand,” Ryan says.

By the end of 2007, Ryan was making hundreds of thousands of dollars a year building systems to make stock-market trades faster. He was struck, over and over again, by how little those he helped understood the technology they were using. Beyond that, he didn’t even really know much about his clients. The big banks — Goldman Sachs, Citigroup — everyone had heard of. Others — Citadel, Getco — were famous on a small scale. He learned that some of these firms were hedge funds, which meant they took money from outside investors. But most of them were proprietary firms, or prop shops, trading only their own founders’ money. A huge number of the outfits he dealt with — Hudson River Trading, Eagle Seven, Simplex Investments, Evolution Financial Technologies, Cooperfund, DRW — no one had ever heard of, and the firms obviously intended to keep it that way. The prop shops were especially strange, because they were both transient and prosperous. “They’d be just five guys in a room. All of them geeks. The leader of each five-man pack is just an arrogant version of that geek.” One day a prop shop was trading; the next, it closed, and all the people in it moved on to work for some big Wall Street bank. One group of guys Ryan saw over and over: four Russian, one Chinese. The arrogant Russian guy, clearly the leader, was named Vladimir, and he and his boys bounced from prop shop to big bank and back to prop shop, writing the computer code that made the actual stock-market trading decisions, which made high-frequency trading possible. Ryan watched them meet with one of the most senior guys at a big Wall Street bank that hoped to employ them — and the Wall Street big shot sucked up to them. “He walks into the meeting and says, ‘I’m always the most important man in the room, but in this case, Vladimir is.’ ”

“I needed someone from the industry to verify that what I was saying was real,” Katsuyama says. He needed, specifically, someone from deep inside the world of high-frequency trading. He spent the better part of a year cold-calling strangers in search of a high-frequency-trading strategist willing to defect. He now suspected that every human being who knew how high-frequency traders made money was making too much money doing it to stop and explain what they were doing. He needed to find another way in.

In the fall of 2009, Katsuyama’s friend at Deutsche Bank mentioned this Irish guy who seemed to be the world’s expert at helping the world’s fastest stock-market traders be faster. Katsuyama called Ronan Ryan and invited him to interview for a job on the RBC trading floor. In his interview, Ryan described what he witnessed inside the exchanges: The frantic competition for nanoseconds, clients’ trying to get their machines closer to the servers within the exchanges, the tens of millions being spent by high-frequency traders for tiny increments of speed. The U.S. stock market was now a class system of haves and have-nots, only what was had was not money but speed (which led to money). The haves paid for nanoseconds; the have-nots had no idea that a nanosecond had value. The haves enjoyed a perfect view of the market; the have-nots never saw the market at all. “I learned more from talking to him in an hour than I learned from six months of reading about [high-frequency trading],” Katsuyama says. “The second I met him, I wanted to hire him.”

He wanted to hire him without being able to fully explain, to his bosses or even to Ryan, what he wanted to hire him for. He couldn’t very well call him vice president in charge of explaining to my clueless superiors why high-frequency trading is a travesty. So he called him a high-frequency-trading strategist. And Ryan finally landed his job on a Wall Street trading floor.

Katsuyama and his team were having trouble turning Thor into a product RBC could sell to investors. They had no control over the path the signals took to get to the exchanges or how much traffic was on the network. Sometimes it took four milliseconds for their orders to arrive at the New York Stock Exchange; other times, it took seven milliseconds. In short, Thor was inconsistent — because, Ryan explained, the paths the electronic signals took from Katsuyama’s desk to the various exchanges were inconsistent. The signal sent from Katsuyama’s desk arrived at the New Jersey exchanges at different times because some exchanges were farther from Katsuyama’s desk than others. The fastest any high-speed trader’s signal could travel from the first exchange it reached to the last one was 465 microseconds, or one two-hundredth of the time it takes to blink. (A microsecond is a millionth of a second.) That is, for Katsuyama’s trading orders to interact with the market as displayed on his trading screens, they needed to arrive at all the exchanges within a 465-microsecond window.

To make his point, Ryan brought in oversize maps of New Jersey showing the fiber-optic networks built by telecom companies. The maps told a story: Any trading signal that originated in Lower Manhattan traveled up the West Side Highway and out the Lincoln Tunnel. Perched immediately outside the tunnel, in Weehawken, N.J., was the BATS exchange. From BATS the routes became more complicated, as they had to find their way through the clutter of the Jersey suburbs. “New Jersey is now carved up like a Thanksgiving turkey,” Ryan says. One way or another, they traveled west to Secaucus, the location of the Direct Edge family of exchanges owned in part by Goldman Sachs and Citadel, and south to the Nasdaq family of exchanges in Carteret. The New York Stock Exchange, less than a mile from Katsuyama’s desk, appeared to be the stock market closest to him — but Ryan’s maps showed the incredible indirection of fiber-optic cable in Manhattan. “To get from Liberty Plaza to 55 Water Street, you might go through Brooklyn,” he explained. “You can go 50 miles to get from Midtown to Downtown. To get from a building to a building across the street, you could travel 15 miles.”

To Katsuyama the maps explained, among other things, why the market on BATS had proved so accurate. The reason they were always able to buy or sell 100 percent of the shares listed on BATS was that BATS was always the first stock market to receive their orders. News of their buying and selling hadn’t had time to spread throughout the marketplace. Inside BATS, high-frequency-trading firms were waiting for news that they could use to trade on the other exchanges. BATS, unsurprisingly, had been created by high-frequency traders.

Eventually Brad Katsuyama came to realize that the most sophisticated investors didn’t know what was going on in their own market. Not the big mutual funds, Fidelity and Vanguard. Not the big money-management firms like T. Rowe Price and Capital Group. Not even the most sophisticated hedge funds. The legendary investor David Einhorn, for instance, was shocked; so was Dan Loeb, another prominent hedge-fund manager. Bill Ackman runs a famous hedge fund, Pershing Square, that often buys large chunks of companies. In the two years before Katsuyama turned up in his office to explain what was happening, Ackman had started to suspect that people might be using the information about his trades to trade ahead of him. “I felt that there was a leak every time,” Ackman says. “I thought maybe it was the prime broker. It wasn’t the kind of leak that I thought.” A salesman at RBC who marketed Thor recalls one big investor calling to say, “You know, I thought I knew what I did for a living, but apparently not, because I had no idea this was going on.”

Katsuyama and Ryan between them met with roughly 500 professional stock-market investors who controlled many trillions of dollars in assets. Most of them had the same reaction: They knew something was very wrong, but they didn’t know what, and now that they knew, they were outraged. Vincent Daniel, a partner at Seawolf, took a long look at this unlikely pair — an Asian-Canadian guy from a bank no one cared about and an Irish guy who was doing a fair impression of a Dublin handyman — who just told him the most incredible true story he had ever heard and said, “Your biggest competitive advantage is that you don’t want to [expletive] me.”

Trust on Wall Street was still — just — possible. The big investors who trusted Katsuyama began to share whatever information they could get their hands on from their other brokers. For instance, several demanded to know from their other Wall Street brokers what percentage of the trades executed on their behalf were executed inside the brokers’ dark pools. Goldman Sachs and Credit Suisse ran the most prominent dark pools, but every brokerage firm strongly encouraged investors who wanted to buy or sell big chunks of stock to do so in that firm’s dark pool. In theory, the brokers were meant to find the best price for their customers. If the customer wanted to buy shares in Chevron, and the best price happened to be on the New York Stock Exchange, the broker was not supposed to stick the customer with a worse price inside its own dark pool. But the dark pools were opaque. Their rules were not published. No outsider could see what went on inside them. It was entirely possible that a broker’s own traders were trading against the customers in its dark pool: There were no rules against doing that. And while the brokers often protested that there were no conflicts of interest inside their dark pools, all the dark pools exhibited the same strange property: A huge percentage of the customer orders sent into a dark pool were executed inside the pool. Even giant investors simply had to take it on faith that Goldman Sachs or Merrill Lynch acted in their interests, despite the obvious financial incentives not to do so. As Mike Gitlin of T. Rowe Price says: “It’s just very hard to prove that any broker dealer is routing the trades to someplace other than the place that is best for you. You couldn’t see what any given broker was doing.” If an investor as large as T. Rowe Price, which acted on behalf of millions of investors, had trouble obtaining the information it needed to determine if its brokers had acted in their interest, what chance did the little guy have?

The deep problem with the system was a kind of moral inertia. So long as it served the narrow self-interests of everyone inside it, no one on the inside would ever seek to change it, no matter how corrupt or sinister it became — though even to use words like “corrupt” and “sinister” made serious people uncomfortable, so Katsuyama avoided them. Maybe his biggest concern, when he spoke to investors, was that he’d be seen as just another nut with a conspiracy theory. One compliment that made him happiest was when a big investor said, “Thank God, finally there’s someone who knows something about high-frequency trading who isn’t an Area 51 guy.” It took him a while to figure out that fate and circumstance had created for him a dramatic role, which he was obliged to play. One night he turned to his wife, Ashley, and said: “It feels like I’m an expert in something that badly needs to be changed. I think there’s only a few people in the world who can do anything about this. If I don’t do something right now — me, Brad Katsuyama — there’s no one to call.”

In May 2011, the small team Katsuyama created — Ronan Ryan, Rob Park and a couple of others — sat around a table in his office, surrounded by the applications of past winners of The Wall Street Journal’s Technology Innovation Awards. RBC’s marketing department had informed them of the awards the day before submissions were due and suggested they put themselves forward — so they were scrambling to figure out which of several categories they belonged in and how to make Thor sound life-changing. “There were papers everywhere,” Park says. “No one sounded like us. There were people who had, like, cured cancer.”

“It was stupid,” Katsuyama says, “there wasn’t even a category to put us into. I think we ended up applying under Other.”

With the purposelessness of the exercise hanging in the air, Park said, “I just had a sick idea.” His idea was to license the technology to one of the exchanges. The line between Wall Street brokers and exchanges had blurred. For a few years, the big Wall Street banks had been running their own private exchanges. The stock exchanges, for their part, were making a bid (that ultimately failed) to become brokers. The bigger ones now offered a service that enabled brokers to simply hand them their stock-market orders, which they would then route — to their own exchange, of course, but also to other exchanges. The service was used mainly by small regional brokerage firms that didn’t have their own routers, but this brokeragelike service opened up, at least in Park’s mind, a new possibility. If just one exchange was handed the tool for protecting investors from market predators, the small brokers from around the country might flock to it, and it might become the mother of all exchanges.

“We just sat there for a while,” Park says, “kind of staring at each other. Create your own stock exchange. What does that even mean?”

A few weeks later Katsuyama flew to Canada and tried to sell his bosses on the idea of an RBC-led stock exchange. Then in the fall of 2011, he canvassed a handful of the world’s biggest money managers (Capital Group, T. Rowe Price, BlackRock, Wellington, Southeastern Asset Management) and some of the most influential hedge funds (run by David Einhorn, Bill Ackman, Daniel Loeb). They all had the same reaction. They loved the idea of a stock exchange that protected investors from Wall Street’s predators. They also thought that for a new stock exchange to be credibly independent of Wall Street, it could not be created by a Wall Street bank. Not even a bank as nice as RBC. If Katsuyama wanted to create the mother of all stock exchanges, he would need to quit his job and do it on his own.

The challenges were obvious. He would need to find money. He would need to persuade a lot of highly paid people to quit their Wall Street jobs to work for tiny fractions of their current salaries — and possibly even supply the capital to pay themselves to work. “I was asking: Can I get the people I need? How long can we survive without getting paid? Will our significant others let us do this?” They did, and Katsuyama’s team followed him to his new venture.

But he also needed to find out if the nine big Wall Street banks that controlled nearly 70 percent of all investor orders would be willing to send those orders to a truly safe exchange. It would be far more difficult to start an exchange premised on fairness if the banks that controlled a vast majority of the customers’ orders were not committed to fairness themselves.

Back in 2008, when it first occurred to Brad Katsuyama that the stock market had become a black box whose inner workings eluded ordinary human understanding, he went looking for technologically gifted people who might help him open the box and understand its contents. He’d started with Rob Park and Ronan Ryan, then others. One was a 20-year-old Stanford junior named Dan Aisen, whose résumé Katsuyama discovered in a pile at RBC. The line that leapt out at him was “Winner of Microsoft’s College Puzzle Challenge.” Every year, Microsoft sponsored this one-day, 10-hour national brain-twisting marathon. It attracted more than a thousand young math and computer-science types. Aisen and three friends competed in 2007 and won the whole thing. “It’s kind of a mix of cryptography, ciphers and Sudoku,” Aisen says. To be really good at it, a person needed not only technical skill but also exceptional pattern recognition. “There’s some element of mechanical work and some element of ‘Aha!’ ” Aisen says. Katsuyama gave Aisen both a job and a nickname, the Puzzle Master, soon shortened, by RBC’s traders, to Puz. Puz was one of the people who had helped create Thor.

Puz’s peculiar ability to solve puzzles was suddenly even more relevant. Creating a new stock exchange is a bit like creating a casino: Its creator needs to ensure that the casino cannot in some way be exploited by the patrons. Or at worst, he needs to know exactly how his system might be gamed, so that he might monitor the exploitation — as a casino follows card counters at the blackjack tables. “You are designing a system,” Puz says, “and you don’t want the system to be gameable.” The trouble with the stock market — with all of the public and private exchanges — was that they were fantastically gameable, and had been gamed: first by clever guys in small shops, and then by prop traders who moved inside the big Wall Street banks. That was the problem, Puz thought. From the point of view of the most sophisticated traders, the stock market wasn’t a mechanism for channeling capital to productive enterprise but a puzzle to be solved. “Investing shouldn’t be about gaming a system,” he says. “It should be about something else.”

The simplest way to design a stock exchange that could not be gamed was to hire the very people best able to game it and encourage them to take their best shots. Katsuyama didn’t know any other national puzzle champions, but Puz did. The only problem was that none of them had ever worked inside a stock exchange. “The Puzzle Masters needed a guide,” Katsuyama says.

Enter Constantine Sokoloff, who had helped build Nasdaq’s matching engine — the computer that matched buyers and sellers. Sokoloff was Russian, born and raised in a city on the Volga River. He had an explanation for why so many of his countrymen wound up in high-frequency trading. The old Soviet educational system channeled people into math and science. And the Soviet-controlled economy was horrible and complicated but riddled with loopholes, an environment that left those who mastered it well prepared for Wall Street in the early 21st century. “We had this system for 70 years,” Sokoloff says. “The more you cultivate a class of people who know how to work around the system, the more people you will have who know how to do it well.”

The Puzzle Masters might not have thought of it this way at first, but in trying to design their exchange so that investors who came to it would remain safe from predatory traders, they were also divining the ways in which high-frequency traders stalked their prey. For example, these traders had helped the public stock exchanges create all sorts of complicated “order types,” The New York Stock Exchange, for one, had created an order type that traded only if the order on the other side of it was smaller than itself — the purpose of which seemed to be to protect high-frequency traders from buying a small number of shares from an investor who was about to depress the market in these shares with a huge sale.

As they worked through the order types, the Puzzle Masters created a taxonomy of predatory behavior in the stock market. Broadly speaking, it appeared as if there were three activities that led to a vast amount of grotesquely unfair trading. The first they called electronic front-running — seeing an investor trying to do something in one place and racing ahead of him to the next (what had happened to Katsuyama when he traded at RBC). The second they called rebate arbitrage — using the new complexity to game the seizing of whatever legal kickbacks, called rebates within the industry, the exchange offered without actually providing the liquidity that the rebate was presumably meant to entice. The third, and probably by far the most widespread, they called slow-market arbitrage. This occurred when a high-frequency trader was able to see the price of a stock change on one exchange and pick off orders sitting on other exchanges before those exchanges were able to react. This happened all day, every day, and very likely generated more billions of dollars a year than the other strategies combined.

All three predatory strategies depended on speed. It was Katsuyama who had the crude first idea to counter them: Everyone was fighting to get in as close to the exchange as possible — why not push them as far away as possible? Put ourselves at a distance, but don’t let anyone else be there. The idea was to locate their exchange’s matching engine at some meaningful distance from the place traders connected to the exchange (called the point of presence) and to require anyone who wanted to trade to connect to the exchange at that point of presence. If you placed every participant in the market far enough away from the exchange, you could eliminate most, and maybe all, of the advantages created by speed. Their matching engine, they already knew, would be located in Weehawken (where they’d been offered cheap space in a data center). The only question was: Where to put the point of presence? “Let’s put it in Nebraska,” someone said, but they all knew it would be harder to get the already reluctant Wall Street banks to connect to their market if the banks had to send people to Omaha to do it. Actually, though, it wasn’t necessary for anyone to move to Nebraska. The delay needed only to be long enough for their new exchange, once it executed some part of a customer’s buy order, to beat high-frequency traders in a race to the shares at any other exchange — that is, to prevent electronic front-running. The necessary delay turned out to be 320 microseconds; that was the time it took them, in the worst case, to send a signal to the exchange farthest from them, the New York Stock Exchange in Mahwah. Just to be sure, they rounded it up to 350 microseconds.

To create the 350-microsecond delay, they needed to keep the new exchange roughly 38 miles from the place the brokers were allowed to connect to the exchange. That was a problem. Having cut one very good deal to put the exchange in Weehawken, they were offered another: to establish the point of presence in a data center in Secaucus. The two data centers were less than 10 miles apart and already populated by other stock exchanges and all the high-frequency stock-market traders. (“We’re going into the lion’s den,” Ryan said.) A bright idea came from a new employee, James Cape, who had just joined them from a high-frequency-trading firm: Coil the fiber. Instead of running straight fiber between the two places, why not coil 38 miles of fiber and stick it in a compartment the size of a shoe box to simulate the effects of the distance. And that’s what they did.

Creating fairness was remarkably simple. They would not sell to any one trader or investor the right to put his computers next to the exchange or special access to data from the exchange. They would pay no rebates to brokers or banks that sent orders; instead, they would charge both sides of any trade the same amount: nine one-hundredths of a cent per share (known as nine mils). They’d allow just three order types: market, limit and Mid-Point Peg, which meant that the investor’s order rested in between the current bid and offer of any stock. If the shares of Procter & Gamble were quoted in the wider market at 80–80.02 (you can buy at $80.02 or sell at $80), a Mid-Point Peg order would trade only at $80.01. “It’s kind of like the fair price,” Katsuyama says.

Finally, to ensure that their own incentives remained as closely aligned as they could be with those of investors, the new exchange did not allow anyone who could trade directly on it to own any piece of it: Its owners were all ordinary investors who needed first to hand their orders to brokers.

But the big Wall Street banks that controlled a majority of all stock-market investor orders played a more complicated role than an online broker like TD Ameritrade. The Wall Street banks controlled not only the orders, and the informational value of those orders, but also dark pools in which those orders might be executed. The banks took different approaches to milking the value of their customers’ orders. All of them tended to send the orders first to their own dark pools before routing them out to the wider market. Inside the dark pool, the bank could trade against the orders itself; or it could sell special access to the dark pool to high-frequency traders. Either way, the value of the customers’ orders was monetized — by the big Wall Street bank for the big Wall Street bank. If the bank was unable to execute an order in its own dark pool, the bank could direct that order first to the exchange that paid the biggest rebate for it.

If the Puzzle Masters were right, and the design of their new exchange eliminated the advantage of speed, it would reduce the informational value of investors’ stock-market orders to zero. If those orders couldn’t be exploited on this new exchange — if the information they contained about investors’ trading intentions was worthless — who would pay for the right to execute them? The big Wall Street banks and online brokers that routed investors’ stock market orders to the new exchange would surrender billions of dollars in revenues in the process. And that, as everyone involved understood, wouldn’t happen without a fight.

Their new exchange needed a name. They called it the Investors Exchange, which wound up being shortened to IEX. Before it opened, on Oct. 25, 2013, the 32 employees of IEX made private guesses as to how many shares they would trade the first day and the first week. The median of the estimates came in at 159,500 shares the first day and 2.75 million shares the first week. The lowest estimate came from the only one among them who had ever built a new stock market from scratch: 2,500 shares the first day and 100,000 the first week. Of the nearly 100 banks and stockbrokerage firms in various stages of agreeing to connect to IEX, most of them small outfits, only about 15 were ready on the first day. Katsuyama guessed, or perhaps hoped, that the exchange would trade between 40 and 50 million shares a day by the end of the first year — that’s about what IEX needed to trade to cover its running costs. If it failed to do that, there was a question of how long it could last. Katsuyama thought that their bid to create an example of a fair financial market — and maybe change Wall Street’s culture — could take more than a year. And, he said, “It’s over when we run out of money.”

On the first day, IEX traded 568,524 shares. Most of the volume came from regional brokerage firms and Wall Street brokers that had no dark pools — RBC and Sanford C. Bernstein. The first week, IEX traded a bit over 12 million shares. Each week after that, the volume grew slightly, until, in the third week of December, IEX was trading roughly 50 million shares each week. On Wednesday, Dec. 18, it traded 11,827,232 shares. But IEX still wasn’t attracting many orders from the big banks. Goldman Sachs, for example, had connected to the exchange, but its orders were arriving in tiny lot sizes, resting for just a few seconds, then leaving.

The first different-looking stock-market order sent by Goldman to IEX landed on Dec. 19, 2013, at 3:09.42 p.m. 662 milliseconds 361 microseconds 406 nanoseconds. Anyone who was in IEX’s one-room office when it arrived would have known that something unusual was happening. The computer screens jitterbugged as the information flowed into the market in an entirely new way — lingering there long enough to trade. One by one, the employees arose from their chairs. Then they began to shout.

“We’re at 15 million!” someone yelled, 10 minutes into the surge. In the previous 331 minutes they had traded roughly 14 million shares.

“Twenty million!”

“Thirty million!”

“We just passed AMEX,” shouted John Schwall, their chief financial officer, referring to the American Stock Exchange. “We’re ahead of AMEX in market share.”

“And we gave them a 120-year head start,” Ryan said, playing a little loose with history. Someone had given him a $300 bottle of Champagne. He’d told Schwall that it was worth only $100, because Schwall didn’t want anyone inside IEX accepting gifts worth more than that from outsiders. Now Ryan fished the contraband from under his desk and found some paper cups.

Someone put down a phone and said, “That was J. P. Morgan, asking, ‘What just happened?’ They say they may have to do something.”

Someone else put down a phone. “That was Goldman. They say they aren’t even big. They’re coming big tomorrow.”

J. P. Morgan, in other words, might actually route investors’ trades to IEX, and Goldman might route more of them than they had done so far.

“Forty million!”

Fifty-one minutes after Goldman Sachs gave them their first honest shot at Wall Street customers’ stock-market orders, the U.S. stock market closed. Katsuyama walked off the floor and into a small office, enclosed by glass. He thought through what had just happened. “We needed one person to buy in and say, ‘You’re right,’ ” he said. “It means that Goldman Sachs agrees with us. Now the others can’t ignore this. They can’t marginalize it.” Then he blinked and said, “I could [expletive] cry now.”

He’d just been given a glimpse of the future — he felt certain of it. If Goldman Sachs was willing to acknowledge to investors that this new market was the best chance for fairness and stability, the other banks would be pressured to follow. The more orders that flowed onto IEX, the better the experience for investors and the harder it would be for the banks to evade this new, fairer market.

IEX had made its point: That to function properly, a financial market didn’t need to be rigged in someone’s favor. It didn’t need payment for order flow and co-location and all sorts of unfair advantages possessed by a small handful of traders. All it needed was for investors to take responsibility for understanding it, and then to seize its controls. “The backbone of the market,” Katsuyama says, “is investors coming together to trade.”

Three weeks later, two months after IEX opened for business, 14 men — the chief executives or the head traders of some of the world’s biggest money managers — gathered in a conference room on top of a Manhattan skyscraper. Together they controlled roughly $2.6 trillion in stock market investments, or about 20 percent of the U.S. market. They had flown in from around the country to hear Katsuyama describe what he learned about the U.S. stock market since IEX opened for trading. “This is the perfect seat to figure all this out,” Katsuyama said. “It’s not like you can stand outside and watch. We had to be in the game to see it.”

What he had discovered was just how badly the market wanted to remain in the shadows. Despite Goldman’s activity, many of the big banks were not following the instructions from investors, their customers, to send orders to IEX. A few of the investors in the room knew this; the rest now learned as much. One of them said: “When we told them we wanted to route to IEX, they said: ‘Why would you want that? We can’t do that!’ ” After the first six weeks of IEX’s life, a big Wall Street bank inadvertently disclosed to one big investor that it hadn’t routed a single order to IEX — despite explicit directions from the investor to do so. Another big mutual fund manager estimated that when he told the big banks to route to IEX, they had done so “at most 10 percent of the time.” A fourth investor was told by three different banks that they didn’t want to connect to IEX because they didn’t want to pay their vendors the $300-a-month connection fee.

Other banks were mostly passive-aggressive, but there were occasions when they became simply aggressive. Katsuyama heard that Credit Suisse employees had spread rumors that IEX wasn’t actually independent but owned by RBC — and thus just a tool of a big bank. He also heard what the big Wall Street banks were already saying to investors to dissuade them from sending orders to IEX: The 350-microsecond delay IEX had introduced to foil the stock-market predator made IEX too slow.

Soon after it opened for trading, IEX published statistics to describe, in a general way, what was happening in its market. Despite the best efforts of Wall Street banks, the average size of IEX’s trades was by far the biggest of any stock exchange, public or private. Trades on IEX were also four times as likely as those elsewhere to trade at the midpoint between the current market bid and offer — which is to say, the price that most would agree was fair. Despite the reluctance of the big Wall Street banks to send orders to IEX, the new exchange was already making the dark pools and public exchanges look bad, even by their own screwed-up standards.

Katsuyama opened the floor for questions.

“Do you think of [high-frequency traders] differently than you did before you opened?” someone asked.

“I hate them a lot less than before we started,” Katsuyama said. “This is not their fault. I think most of them have just rationalized that the market is creating the inefficiencies, and they are just capitalizing on them. Really it’s brilliant what they have done within the bounds of the regulation. They are much less of a villain than I thought. The system has let down the investor.”

“How many good brokers are there?” asked an investor.

“Ten,” Katsuyama said. (IEX had dealings with 94.) The 10 included RBC, Bernstein and a bunch of even smaller outfits that seemed to be acting in the best interests of their investors. “Three are meaningful,” he added: Morgan Stanley, J. P. Morgan and Goldman Sachs.

One investor asked, “Why would any broker behave well?”

“The long-term benefit is that when the [expletive] hits the fan, it will quickly become clear who made good decisions and who made bad decisions,” Katsuyama said. In other words, when some future stock-market crash happened, perhaps a result of the market’s technological complexity, the big Wall Street banks would get the blame.

The stock market really was rigged. Katsuyama often wondered how enterprising politicians and plaintiffs’ lawyers and state attorneys general would respond to that realization. (This March, the New York attorney general, Eric Schneiderman, announced a new investigation of the stock exchanges and the dark pools, and their relationships with high-frequency traders. Not long after, the president of Goldman Sachs, Gary Cohn, published an op-ed in The Wall Street Journal, saying that Goldman wanted nothing to do with the bad things happening in the stock market.) The thought of going after those who profited didn’t give Katsuyama all that much pleasure. He just wanted to fix the problem. At some level, he still didn’t understand why some Wall Street banks needed to make his task so difficult.

Technology had collided with Wall Street in a peculiar way. It had been used to increase efficiency. But it had also been used to introduce a peculiar sort of market inefficiency. Taking advantage of loopholes in some well-meaning regulation introduced in the mid-2000s, some large amount of what Wall Street had been doing with technology was simply so someone inside the financial markets would know something that the outside world did not. The same system that once gave us subprime-mortgage collateralized debt obligations no investor could possibly truly understand now gave us stock-market trades involving fractions of a penny that occurred at unsafe speeds using order types that no investor could possibly truly understand. That is why Brad Katsuyama’s desire to explain things so that others would understand was so seditious. He attacked the newly automated financial system at its core, where the money was made from its incomprehensibility.

The end may be near for HFT and the bull market. If you have concentrated your wealth too much into equities (and bonds, real estate too), then you should consider alternatives. CASH itself may be safer. Equities are still near all time highs.

Cliff notes: Unless you literally own the technological AND physical infrastructure that allows you total & complete privacy & dominant speed in EXECUTING TRADES, in the "new normal" ""STOCK MARKET,"''YOU ARE DELUSIONAL TO THINK YOU CAN BE ANYTHING OTHER THAN PREY, TO BE SWALLOWED QUICKLY & ENTIRELY.

The biggest theft in all of human history, leading to wealth inequality greater than at any time since the 1920s, all enabled by a more sophisticated & technologically advanced form of plain theft & fraud - aided & abetted by FRACTIONAL FIAT RESERVE CENTRAL "BANKS" and bought & bribed REVOLVING DOOR REGULATORY AGENCIES.

What's being disclosed is the tip of the iceberg, no doubt. The type of activity described and amount of illicit gains enabled by it is but one of many fraudulent ways the sheep are sheared 24/7/30/365.

I have worked in applied technology for almost 20 years. I realized long ago that a tool is neither good, nor evil; it is (wo)men who are good or evil (had to include Blythe). What we are living through is not any different, in principle, than all the other frauds and shenanigans throughout history.

Our species is designed to survive, in good times and bad, so don't expect too much in the long run.

Lastly, at the current rate we are breeding and consuming finite energy resources, we will be defrauding each other of sea shells and plow mules in one hundred years. At least we will be less of a threat to all the other life on Earth by then, and likely just as happy/miserable.

How does the exponential growth of human population reflect in your figures? how does the current use or lack of desire to use this form of energy by governments and corporations reflect the current problems we face?

If anyone can think, we can just switch to any of this energy as the main consumption globally overnight, then they are seriously dillusional.

What we are using NOW as main consumption is finite and it will get more expensive the more time we fuck around not diversing into proper clean energy.

"If anyone can think, we can just switch to any of this energy as the main consumption globally overnight, then they are seriously dillusional."

Yeah they are. But surely the known reserves of coal, oil, uranium, thorium, natural gas, etc. are not going to run out overnight...

The energy alarmists are almost as stupid as the global warming alarmists. Just because your infinite stupidity cannot see a possible future beyond a 100 years, doesn't mean that noone else can see a future.

The amount of ways we can adapt are so numerous that I find it a total insult to listen to people saying that humans are that hopeless that they can't get enough energy to sustain themselves. If you're that pessimistic then why not stop eating now, just give up.

good read. I've worked a little with the Juniper gear, but the Arista switches maintain the fastest throughput. Finally, a description easy enough for one who holds public office to understand (..although irrelevant as they generally choose to ignore criminal behaviour)

What's happening is that all the easy to get stuff is already out, plus most of the not so easy. Eventually what will happen is that all which is left are the reserves which are so hard to get out that the energy spent in getting them out is less than the energy contained in the fuel itself.

That said, even this is not one of the major problems. The biggest problems are similar to the way companies which have more assets than liabilities can still go bankrupt because for a period their cash-flow is negative or can over time fail because their revenues dropped below their expenses:

- There might be a ton of other energy options out there but our technology to get at them is not efficient enough and the infrastructure is not in place yet. If the supplies of cheap to extract fossile fuels drop faster than the technology and infractructure to generate and use renewable energy improves, then we risk a long period where the revenues (energy extracted minus cost of extracting energy) are less than the expenses (energy spent by our society). If this period lasts long enough, society as we know with might sink enough that we cannot sustain the necessary technological development or the infrastructure costs to have enough renewable energy generation to fully power our society.

This is the thing which the drill-baby-drill crown doesn't seem to understand: researching and investing in alternative energy generation technology now is just good business sense, in the same way as a well-managed company invests in research of new products WHILE their existing products are still producing profits, so as to avoid going bankrupt by having no new products in the pipeline when the old products stop bringing enough revenue.

The reason for the 'exponential growth' of the human population is we've removed a lot of early death from the equation. Malaria, TB, polio, smallpox, etc. It's not that more and more children are being born; it's that more and more of them are surviving into their sixties.

The tendency, as populations increase and get more educated, is for women to have smaller families. Already, in parts of Europe and America, the white people are actually losing population, as their women won't have children at all. Best estimates right now are that world population will stablize around 9 billion. Still a lot of people, and there are always problems of distribution to be solved, but we're not going to have 100 billion people on the planet, ever.

In your infinte stupidity, you forget (or never considered) that energy, technology, economy and finance are all interconnected, interdependent, and intimately involved in complicated symiotic relationships.

Simply put, if one aspect of this symbiotic system suffers a decline or reversal, the rest of the system degrades, and that degradation leads to further reversals and slippages. Once the downward spiral dynamic is begun, it reinforces itself, until it gains enough momentum to become unstoppable.

For example, how does one take uranium ore embedded in a rock in a hillside, and convert it into electricity?

Answer: Through an incredibly complicated industrial process that requires not only sophisticated technology, but an entire ecosystem of supporting industry that has a broad base in heavy manufacturing, ie, steel, sulfuric acid, coke, etc. This in turn needs a sophisticated transportation system to move heavy raw materials. All of this is built and maintained by a functional system of finance and economy.

If any of this broad-based, interdependent system begins to break down, then uranium becomes unusable. So, where is the weak link? What cheap, infintely scaleable, easily transportable, massively flexible and extremely versatile resource is needed to make all the other processes possible?

Petroleum.

But even more to the point, oil drives the financial markets, in that, as THE basic precursor for an industrialized economy, any decline in oil production will be magnified, as any decline in oil production will equal a decline in the ability to repay debt.

In a debt-based economy, then, a decline in oil production will equal a drop in liquidity. This liquidity squeeze will be magnified in the financial markets. Thus, any small but real percentage drop in oil production, in consideration of today's massive debt and leverage overhangs, has the potential to crash our current financial system.

So, no money means a sieze-up of the industrial economy. Without access to credit or cash-flow, coal mines shut down, barge companies stop running, chemical factories close, shortages of raw materials prevent high technology companies from being able to manufacture everything from servos to microchips to carbon-fiber composites. Without all this integration, let's see...

Admittedly I tend to be skeptical and cynical, so take it with a grain of salt when I say that if HFT gets taken down it will not be because it was unfair to the average investor or trader, but because the big banks realized they were beginning to focus on HFT in a way that resulted in them parasitizing each other, and not focusing enough on their work-together-to-shear-the-people cartel status under the Fed.

That was awesome, read the entire thing. I don't see why trading isn't settled say ever second, orders are either in or out at an agreed settlement time meaning you can't pull or place an offer until the prior to the next second settlement time. Trades could easily only match and clear at specific intervals, not versus any old time. Essentially thats what these guys did in a roundabout way. Still seems to simple to solve through regulation, slow things down, create set trade settlement intervals, problem solved. I guess that makes too much sense.

Well, let's see, there is no market incentive, and there are no regulators with the brains to understand it nor the balls to enforce it. Ergo, IEX. A computer-driven market solution. Capitalism doesn't appear completely broken yet. This is only the first chapter of the IEX story, though.

in short, Greed (and the laws of math) are why things are the way they are. And they always will be... just look at all the greed and envy that manifests itself as generic derision for the 1% (or the .0001%). Greed will never go away.

And when it all goes BOOM! (like before I've had my morning coffee and my brain starts working), they'll just start work on the next skim (unfortunately if they have control of both the .gov and the banks) then the bankers might impart their knowledge of the operation of skims to the .govs and we get the ultimate evil -- monopoly digital currency with an FTT that targets those not in the .0001%.

(edit: the money supply is not at rest but it works for both momentum and the full equation, but I can't input integrals in ascii)

They'll do what they always do: Shake a stick at 'em, make some noise, extract a bribe of some sorts, pass some new grandiose legislation that is supposed to "fix" the issue, but really enslaves the general public/sheeple/muppets even more.

I read the whole article. Brad Katsuyama in a way is kind of like the Canadian Ed Snowden...If not him, who, who would have exposed this scam? He exposed something ZHers already knew about regarding the HFT scam to millions of the mainstream last night.

I actually heard 2 young people in line at Trader Joes tonight talking about the 60 Minutes piece and how the stock market is fixed.

He brought this immoral skimming operation into the bright sunshine and anyone that puts themselves on the line this way deserves all the credit they get.

Let's celebrate a small victory for honesty and integrity in a world full of evil sociopaths, corruption and immorality. Brad Katsuyama represents a small glimmer of hope many of us on ZH harbor deep down. That's one for the good guys.

By the time the average person who is even exposed to the "equity markets" comprehends one one-thousandth of the HFT scam, in any of its permutations, Wall Street will be well onto its next big fleecing criminal enterprise & racketeering operation - wait, how naive of me; given the tip of this iceberg that has ostensibly been "uncovered,' rest assured Wall Street has already moved on to the next Big Heist.

"...rest assured Wall Street has already moved on to the next Big Heist."

That's what Morgan Stanley, J. P. Morgan and Goldman Sachs' involvement with IEX says to me. There is NO WAY they are fighting for truth and goodness; they've already found other ways to game the system and are now trying to hobble their competitors.

It is an unfortunate waste that so much young talent threw itself into the casino over the years. When this shit show stops, it is going to be a tough row to hoe for a lot of these folks finding work that produces value, if it can be found at all.

Broadly speaking, education, health care, real estate, insurance, and finance are all in for a very rude awakening, just like the manufacturing base that got whored out.

What's simply amazing, is how much denial exists on ZH, its like if they just keep posting Santelli, Cramers, Baritomo, and all the CNBC talking heads the good days are coming back.

They're not, ... they fucked most people in DOT-CON (1999), and then fucked them again in 2007, and its never going to ever be again, the way it was, back in the day.

Easy money rarely lasts forever, smart guys see the writing on the wall and find a new gig, before they're living on the streets.

*

HFT is NOT going away unless WASH-DC is released by NYC financial interests, and that isn't going to happen unless the NYC FIAT (USD) quit's being TENDER. When will USD collapse?, when TEL-AVIV flashes the exit-light.

Reminds one of the never-ending river spring break-up pool. Betting on which day in April the false flag event will happen, before the shirt hits the fan and something unexpected that is BEYOND babylon's control happens first. What a quagmire. Who's responsible for this blarney show?

To see the future, one needs only watch the chess players and where they are going. When the rats hit the air, seas, roads, trains and planes, it will be a time to duck and cover. Who was warned not to go to work and to get out of the trade towers by the message company just before 9/11 happened? Who knew building number 7 was going to fall before it happened when he said, "we made the decision to just, pull it."

All hands on deck. Something will break as planed, or not. The crime cats are out of the bag.

I'd flash crash natty gas when the bid disapeared in front month 2008, I'd throw in a one std deviation + a tick bid often half hour before a quiet close; occasionally I'd blink and 15c would appear and I'd immediately sell. Not too long after that I saw what Katsuyama experienced. I remember dragging my no-way finance partner to watch while I would load up an incremental better bid; instantly I was the event and that was the eureka moment. I stopped playing, but more understanding why the free train don't stop here. It only stops in the minds of the escaped ones.

IEX is a joke. In order to execute the trade there has to be the other side of the trade available. If IEX can not provide it, especially given their diminutive size, it must seek a match through another exchange and then, whalla!, those able to see a trade in advance within those exchanges will see the request from IEX and nothing is accomplished. In fact, they will lkely seek to punish trades routed from IEX.

Fascinating stuff.
My reaction to the mention of Gary Cohn is that this is an indicator that Goldman realizes they might be about to be hit by the vengeance of an angry citizenry and they want to get on the right side of public relations ASAP.

And Katsuyama's being disingenuous toward the end of the excerpt. As he realizes but didn't want to say, the system got like this because people at the big banks and the major exchanges are fvcking crooks.

Yep, right 'blame the computers', or more correctly we can fix all the problems by reverting to the old human system, where all trades were handled by 'specialists' and guys on the phone, doing cold call's and fucking grandma and grandpa.

I doubt that anybody on earth, other than a few computer nerds in NYC give one flying fuck about this story.

Not holding my breath, but really hoping Schneiderman does something with this. And if you are reading this, Mr. Attorney-General, heed this advice and don't bang any hookers in the coming weeks, we all know what happened to the last guy who sat in your chair and tried to fuck with Wall Street.

Do we need to read to know that this bitch market is broken. Let me quote stuff all you mother fuckers on wall street and let me book fade and salami slice you pernicious tacky sleazy salty cock sucking mother fucking kykes.

"The online broker TD Ameritrade, for example, was paid hundreds of millions of dollars each year to send its orders to a hedge fund called Citadel, which executed the orders on behalf of TD Ameritrade. Why was Citadel willing to pay so much to see the flow? No one could say with certainty what Citadel’s advantage was."

Any guesses??????

I do not consider Morgan Stanley, J. P. Morgan or Goldman Sachs in any way to represent "the little guy" unless you think the trillions in 401(k) money they manage somehow falls into this category.

The little guys are with brokers like TD Ameritrade, E-Trade and the like and precious little attention is focused on what has happened to those who placed their trust with these brokers. A huge class action lawsuit should be filed based on the quote above alone.

This was a good article. It's made me wonder, strangely enough, if the h.f. trading was eliminated could the market actually go up afterwards ? (rather than crash) Not that stocks don't seem wildly over priced to me (not gold stocks !) but with all the printing being done by the fed, the paper has to go somewhere doesn't it ? Stocks can be better than cash for protection from inflation which may be a larger threat than a stock market crash. A lot of people who would be in the markets aren't because of the perception it's rigged because of hft etc., remove that and maybe cash that is parked will start to return to stocks, seeking better leverage against inflation.

I see HFT and QE as essentially orthogonal. Yes, there are all sorts of bizarre HFT games going on (thanks, Nanex!), but they pale in comparison to the distortions caused by several trillion dollars' worth of fake money.

Funny how people that are awake get the clues that all point to something wicked comes this way again, coming soon.

Of course, we all hope the world's people will all decide to get along, and they will all combine to rid the planet of the moneychangers that want to kill off every one, and the planet.

One just stands in awe...staring up at the tallest article ever seen on this channel. Like, just wow. That was some serial reporting. What an honor to be part of history, and to witness the great works of men and women that use real intelligence to near perfectly shine the light on the real problems at the center of the whole corrupted moving picture show. Questions to monkey men include, how much longer will they continue to support the corruption and criminals at large? Where is that bush gang at now, anyways?

Something came to the fore for a consideration and ponderment. Did they snatch that jet from Malaysia and take it to DG to refuel it and restock it as they told the people they were just saved from a hijacking as they took the scientists that were working on the cloaking device off, and then fly to Pakistan?

And, in the last days of his presidency when bush jr went to Pakistan just after the Minot nukes went missing, did he deliver the nuke under his cover to wait for this plane to arrive years later to become that one false flag event that will really get the hornets nest terrifically excited somewhere? What if, they put that missing Minot thingy on that plane from Malaysia in Pakistan and did something crazy with it to start a war, or something just as stupid?

Now, who could ever do such a thing and who would benefit the most from creating this false flag event? Lest see now, who was on watch when 9/11 happened and who all was involved? Think they aren't still around up to no good after profiting from that evil deed? Where are those bushes at, anyways?

All eyes on Pakistan and on the home fronts. Sure wish we had some of our own troops guarding our borders.

Who is going to instigate another reason to declare moar woar on someone next?

It really is amazing at this point that the rest of the world hasn't all ganged together against America and told us to go shit in our hats. We continue to lay a big fat turd in the punchbowl day after day and getting away with it.

I cannot believe the Hollywood'esque ending though.Morgan Stanley, J. P. Morgan and Goldman Sachs don't need to care what happens when SHTF, as they have bought politicians, regulators, and judges alike.So what is the real reason?

And how cute is it that now after the horses escaped, the barn burnt down, years later someone built a new one and put fresh horses in it, and only now the regulators start worrying about the barndoors?

"If we were in the wild, I would attack you. Even if you weren’t in my food chain, I would go out of my way to attack you! If I were a lion and you were a tuna, I’d swim out in the middle of the ocean and fucking EAT YOU! And then I’d bang your tuna girlfriend."

Another problem I have run into in Comex ... there is a vacancy in the order chain (i.e. no bid or asked at a given price) , I place an order buy or sell, into a vacant slot and instantly my order is doubled; place 1 and 2 contracts appears; two and my two and two others appear. When I cancel lets say one contract the two contracts drop to 0. It's like someone 'knows' I am willing to buy/sell at a given price. This only happens when the slot is blank, i.e. when there are seven and I add one, nothing happens.

Yeah, that's been happening for years. It's not just comex and it's not just the USA. The process is most obvious in a completely empty book, not a single quote in sight, only a few trades all day. Place a quote and zing, there's a bot parked just ahead of you with the same order size, hijacking all the research that may have gone into your quote.

If you cancel, it cancels. If you edit, it edits with you, if you flip sides it flips with you. I made a video in 2009 showing how the hijacking quotes sometimes appeared BEFORE my triggering quotes. My guess was that the bot was operated by the same bank that ran the dark pool into which my orders first went, hence the bot could present a quote to the world before the dark pool released my own.

The algo's aim is to constantly queue-jump, pick up some hits that you should have got, then flip to the other side and offer at a worse price, hoping you'll get impatient and hit it, giving it the spread. Standard "market making" behavior, just a bit ridiculous for such low volume.

The way to game them is to use a large order size and stepwise worsen the quote until you find the point where the bot won't queuejump anymore, backup so it plays then wait for it to take some hits. When it does commit, then back off or cancel then sit at a good price as long as it takes. The bots don't like to hold overnight so they'll often capitulate and pass through the hits at a loss to themselves at the end of the day.

I don't recommend trying to do it manually though, as you'll eventually end up with a full or partial bad fill. I wrote my own algo to do this back when I used to be stupid enough to think I could compete.

The whole thing it a major pain in the ass. A predatory middle man getting in the way of actual trading.

I'm still not quite sure who actually loses money with HFT. The "mom and pop" retail daytrader, sure, they're always ripe for picking, and its no surprise. But if one just buys a stock, puts it away, collects dividends and never trades it, where's the loss?

However, as the poster above noted, if this is being done in derivatives of your securities, you could wind up with say, IBM, that you bought at 80 years ago, saw it go to over 200 in 2008, and then when the herd panics in the derivatives market, the options for IBM, as they did in March 2008, your IBM is now worth 69 and you are, on paper losing 11 points. Only TARP and the FED saved you by pumping it back to 200. If they had not done so, you would be crying the blues having owned the stock for 10 yrs and being down 12%. If you needed that money for retirement expenses, the blow would have been awfully hard to take.

The Reasons for this trump all the possible other contra ones about a crash, bubble and other short prone information you read about.

Those pension funds that own IBM are now whole once again and IBM liquidations to pay out retirement cash, are not putting any downward pressure on the stock or the dividend because another primary dealer is using some of that n billions a month to 'take up' the shares at a few dollars or cents more or less.

"He had an explanation for why so many of his countrymen wound up in high-frequency trading. The old Soviet educational system channeled people into math and science. And the Soviet-controlled economy was horrible and complicated but riddled with loopholes, an environment that left those who mastered it well prepared for Wall Street in the early 21st century. “We had this system for 70 years,” Sokoloff says. “The more you cultivate a class of people who know how to work around the system, the more people you will have who know how to do it well.”"

Another reason, I would submit, is that Russians are willing to work quite cheap, and the financial companies like to hire them on H-1B instead of considering the domestic talent pool. Goes along with the general theme that the financial sector has engaged in, to replace every IT worker possible in their organization with foreigners.

"A bright idea came from a new employee, James Cape, who had just joined them from a high-frequency-trading firm: Coil the fiber. Instead of running straight fiber between the two places, why not coil 38 miles of fiber and stick it in a compartment the size of a shoe box to simulate the effects of the distance."

I can see why HFT elevates the market......I place an at market order to buy 100 shares and HFT sees it en route...buys the 100 shares then replaces them at market 2c higher while my order is still en route and my order gets filled 2c higher than I thought it would. HFT skims the 2c.

But if I placed a sell order this is harder as HFT would probably have to execute it as a naked short .....so the pressure down is less than the pressure up ....so the constant pressure is up.

What I havent seen discussed is whether HFT is being used in the derivates market. I imagine it could and probably is....and here lies the potential leverage for chaotic disaster imo......eg sudden HFT flocking into the buying of a popular put option putting downward pressure on a stock and whoosh...downward leveraged whirlpool sucking all and sundry with it.

Wrong. Given the time between trade and settlement, and factoring in the actual holding period of the HFT, selling is virtually the same as buying. It's not like they have to deliver the equity in a nanosecond. They have plenty of time to borrow from a Prime Broker if need be. HFT is an equal opportunity clipper, upside or downside.

I wouldn't put it past the TPTB to give this story traction in order to have the ADHD public's attention diverted from Jumping Bankers, Missing Gold and a whole assortment of other stories they'd want hushed up.

Talking heads were discussing this on Fox yesterday. Mentioned in the first 5 minutes "front running" and "oldes trick in the book" but were horrified when someone mentioned "rigged markets" that would be absolutely untrue. However everyone polled agreed it needed to be looked into and had concerns it might affect market "liquidity". My ASS! The curtain has been pulled back a bit and the great oz is a huckster. Who would have thunk it? Let me guess, the fed will investigate, it's all over, won't happen again, move along, nothing to see here.

As I have been saying here for a very long time: Just add a random delay to processing of all transactions (0 to 1 second from a uniform distribution) and the HFT issues go away. The algos can't work. "Poof". Problem solved.

Makes me know any money we pay the SEC is a total waste. Otherwise this would have been solved as soon as the scheme was detected (didn't Max Keiser take credit for inventing it?).

The damage inflicted upon individual traders - traders who had the moxie but maybe not the weaponry - was far more fatal financially in the commodity markets. If x was their understanding of downside risk , it no longer was. That risk was now 50 - 100 % greater than previously understood. Some people knew this , but not them.

I have been enjoying the shitshow since the 60 Minutes piece Sunday night. Even CNBS was buzzing about it all day, yestreday and now that the book is out and #1 on Amazon, there will be even more buzz.

I also downvoted some of the more cynical commentators here because I believe this is the beginning of the end of the current greed era. Yes, there will be another one, but maybe for a while there will be honest and open markets, and, at the very least, many people's eyes have been opened.

What most do not realize is that the volume of trading on Wall Street has been in precipitous decline for any number of years, probably since 2008, maybe longer. The computer/internet revolution brought in individual investors. the dotcom crash cleared out a lot of them. 2008-09 wiped out the rest. They're not coming back.

Now, if the SEC would turn off the porn for a bit and quit dragging their feet on writing the rules for the "crowdfunding" bill, we might actually have something with which to run an open and fair society. Surely, there's room for scams in the public sector, but this exposure of the HFTs is a very good start at dismantling the Wall Street oligopoly.

a man walks into a casino every day and bets $1,000 on the green zero on the wheel. After a year of getting ass fucked by the casino, and a $365,000 loss he says to himself "perhaps i should not play in the casino..."

I'm surprised Mr. Katsuyama has not had a one car accident long before this book got published. Way too much money involved in this racket for it to go down quietly. The funny thing is, as others have mentioned, this is one of the least offensive frauds run on Wall St daily

I dealt with a produce company in Detroit while managing 5 large national chain restaurants. I spent over 150k a week with them. I knew they were mob owned but they took good care of us and were suppliers when I began with the company. I made a decision to change produce suppliers when I began getting subpar products and my sales rep ignored my pleas for help. The very next morning at 630 AM as I made my way to the rear entrance of the restaurant there was a black BMW parked in front of my entry door. A man in a custom tailored suit stepped out and introduced himself. His name was on the company trucks and he was son of the owner. "I understand you choose not to business with us?" I looked behind the dumpsters to make sure he was alone. He reached in his jacket and pulled out a business card. He said, "Tomorrow when you get your produce you can call me at this number as he wrote a phone number on back of his business card." "This rings next to my bed on my nightstand." I told him i didn't appreciate being ignored and my company held me liable for quality and product availability on my menu. I told him I didn't appreciate his veiled threat by showing up at my work in the wee hours of the morning and all of thats implications. I said I will give you one more chance but don't ever threaten me again. We shook hands and there was never another quality issue. I figured I had pushed the envelope as far as possible without incurring another threat. The corruption in our country is nothing new. Money changes hands. Names are whispered in dark alleys. Those who won't play go away. It is at all levels of government, politics, finance and business. America has become a whore selling her wares to the highest bidder. It is hard to lose money when you own a casiono or a whore house but maybe sooner or later the whores and the gamblers wake up and realize who they really represent to those in control. Is America at that place yet? I don't know. Police shoot and kill a man in New Mexico from 40 yards who is mentally ill and poses no threats? City of Chicago paid almost 100 million dollars in settlements to police brutality victims in just 2012. It is all broken. Love your families. Keep your freinds close and your enemies closer. Choose your battles wisely because the price of making your stand in this country now is going to be very very costly. I give this man credit for coming out with this information. I hope he is prepared to meet a man in a parking lot at sunrise and he stays aware of his surroundings.

What a great story. Uplifting to see what a Canuck and an Irishman can pull off with a little ingenuity and salt. You see these bastards are not Gods they are actually people who are used to winning a fixed game... they have never had to truly compete, they are incompetents. Maybe the question we should be asking is not if they can be beaten but how often people try.

EDS / Ross Perot wanted to automate WS/NYSE back in 1970... he was told to fuck off... nothing has changed since..... at the time...the big boys wont travel from old greenwich unless they could go home with 10-20 million.... WS is the jungle in a designer suit.. now its at least a billion before they bother... institutional side of MLPFS was telling its clients to sell the exact same stocks the retail side was telling orphans and widows to buy....Penn Central, Grant Dept Stores and other dogs going bankrupt... nice group of people... that's the day i quit...wall street is basically run at the top as a rigged game by dishonest unethical, immoral people willing to do anything to enrich themselves... at the expense of anyone they can... including their mothers.

Now, that stat puts EVERYTHING in perspective. Meaning there is a lot of meat on that "time" bone...lots of margin. Let's say 4 milliseconds for simplicity....that means 25 trades by the time you blink quickly.

This is exactly what I thought was happening. Market Makers add more volume then they want to multiple exchanges. They don't know where the volume is going to occur, so they have to place orders at every single exchange. When someone unintelligently sends a buy order to these exchanges, the orders do not get to each exchange at the same time. Whatever exchange is closest or fastest will give market makers their first fill acknowledgements. When market makers sense a large fill at the close exchange, they send out aggressive orders to take out the other exchanges or simply cancel orders. Because HFT is in a constant competition to be the fastest, they will be able to cancel orders at other exchanges before the initiating order gets there.

The problem is not high frequency. The problem is the fact that stocks are fragmented. Please try buying the order book of the Ten Year Treasury on the CME. I can assure you, you will not get filled on soly the first level.

What I like about the way that Lewis writes, and why I *HOPE* he will take up the coverage of Bitcoin - is how he can take a pretty obscure topic like HFT-acrononsense and in the first couple of paragraphs allow even a dim-Witz like me "Get It". Even encourage "Getting It".