A Software Expert Explains How High Frequency Trading Programs Crash, And How Little We're Doing To Stop Them

All we know when something goes wrong with Wall Street's
super-fast trading programs, is that millions of dollars can be
gone in an instant. In the rush to recover what was lost, it's
very rare that we learn how everything disappeared.

The answer is in the software, some say — the lines and lines of
code written to bring algorithms to life. They have the power to
bring stock markets to a stand still, and in all the discussion
about high frequency trading, they're being completely forgotten.

At least, that's what it seems like to software experts like Lev
Lesokhin. He works for CAST, a company that visualizes
inherent risks in financial software systems. He sees what can go
wrong for his clients when programs are being built, and he says
that regulators are ignoring one of the major problems with high
frequency trading (HFT) programs.

"This is true across industries based on technology. They don't
really... have good oversight of the structure of their
systems... Look under the hood of these companies... there's not
enough attention being paid to the programs they're using, and
these programs are a large part of what's causing the problem."

What the SECis doing, says Lesokhin, is
trying to regulate human traders, not the software that carries
out their wishes.

Take this week for example, the SEC hosted a roundtable to
discuss how stock market shocks like the 2010 Flash Crash and the Knight Capital's trading
glitch can be prevented. By the end of the day, the group
generally agreed that automated 'kill switches' — a program that
would stop a trades dead in their tracks if certain triggers were
hit — would be a good idea.

Here's the problem with that.

"Kill switches are kind of a lipstick on a pig situation," said
Lesokhin. "It might help with some of these software glitch
issues...(but) to me it seems like kind of a no brainer...
it just sounds cool because it's automated.

Why a no brainer? Lesokhin explained that algorithm designers are
smart enough to write code around whatever situation can trigger
the kill switch.

On top of that, there are tons of situations that occur in
markets that can't be predicted or tested.

Take what technically happened to Knight Capital, for example.
Their trading glitch occurred when the company updated its
software (something companies do more and more frequently these
days).

The glitch can be traced to some dead code lying dormant within
the program.

The new update brought that dead code back to life like
Frankenstein. It started creating havoc and spitting out the
trades that cost Knight Capital $440 million.

"Most IT applications have dead code," Leskhin said. "It's in
there just hanging out in the code base but none of the live
modules are calling it. If you don't have structural oversight
then you don't know if your new live code could be calling the
dead code."

Another example of how the unseen (and, in this case, untestable)
can send stocks spinning out of control is what caused a crash at
the NASDAQ during the Facebook
IPO.

To understand it, you have to get a little bit technical. NASDAQ
was written in C++ and there was a "concurrency issue," Lesokhin
explained.

Concurrency is when two processes run parallel to each other, but
ultimately, one triggers action in the other at a certain point.
That trigger, and the action that follows, is what keeps the
program moving forward. It's almost like a relay except the two
guys start running at the same time. At some point the first guy
is supposed to hand the baton to the second guy so he can finish
the race.

At NASDAQ that didn't happen. The baton wasn't passed and the two
processes couldn't move forward. As a result, they started
communicating in an infinite loop, waiting for a signal that
would never come.

The whole mess was caused by an unforeseen combination of
variables coming in from all over the market.

"You can test and test and test that but you'll never run into
problems until you have the wrong combination of data," said
Lesokhin.

In his view, then, that means these programs need to be poked and
prodded by people who understand the software. And the SEC needs
to set standards for vulnerability and structural soundness that
firms have to adhere to.

More disclosure and regulation around their software is likely
the last thing HFT firms would like to see. Companies want
programs to go faster and faster, and they want updates more and
more. The business side puts pressure on the IT side, and the IT
side has interests of its own.

"There's a lot of interest at stake and I don't think anyone
wants to come to the fore and say 'we want to be regulated.' I
don't see tech people talking about this and (when it's time to
talk regulation) most of the voices in the room are trading
experts, not technology experts."

In short you can try to stop exchanges from favoring HFT firms,
you can try to stop algorithms from being predatory all you want,
and you can stop trades dead tracks, but that's not going to stop
a lot of problems with HFT before it's too late.

"The systems are getting too complex for any of the brilliant
developers they have building them to manage them," Lesokhin
added.