What Wall Street Absolutely Must Do To Keep Going Through The Next Disaster

Wall Street was shut down for two days this week, an unprecedented occurrence that only a massive natural disaster like Hurricane Sandy could prompt.

It's a decent excuse, but it's not enough to stop the finger-pointing going on. Some say the NYSE didn't need to shut down.

Others say banks and trading firms wouldn't have been prepared to work even if the NYSE hadn't shut down because they hadn't installed the NYSE's trading system, Arca, as a back-up, in case their trading platforms failed.

To find out what all this actually means for the people who have to pay for these platforms, Business Insider spoke to software expert Lev Lesokhin. He works for CAST, a company that visualizes inherent risks in financial software systems.

In his view, the NYSE trading platform could be a good back-up, but as with all software systems, the firms have to be careful about adding it to the thousands of systems they may already use. Think: That 5,001st system out of the 5,000 a firm already has could be the straw that breaks the camel's back.

That's because every time a new trading platform is integrated into a firm's system, it must be customized to ensure that the new program and the old program play nice together. For one, firms have to make sure that the new program doesn't trigger dead code in the old program.

And there's more:

"You get into all of the typical dangers when you customize," Lesokhin told Business Insider . "With more data flowing you get into performance issues.... With a new system you have another set of transactions that now have to plug into your existing systems... they may slow down your existing systems."

All of this takes time, money and man-power — it can't simply be done over night. Even if a firm's programmers take one day of work to customize on a big complex system, they need to leave weeks for quality assurance and testing.

But even that can leave gaping holes.

"You cannot possibly test your way through...there's so many paths through these systems, so many potential scenarios that it would take years to test and nobody has the time.... most organizations are starting to move towards having some kind of automated oversight."

Automated oversight programs are continuously checking the status of trading platforms making sure that everything is working normally.

All of that is just what it takes to add a back-up trading programs. What Lesokhin truly recommends to prevent the next Wall Street shutdown is that every bank and trading firm have a mirror site in a remote location that maintains an exact duplicate of their trading systems.

After all, in the end, this is all about managing tons of data (trading positions etc.). With mirror sites, you may lose a minute of trading, but that can be recovered.

Banks know they'd be better off with mirror sites (and even more contingency plans) in place. And Lesokhin pointed out that regulators have become more aggressive about seeing this too.

And lets be real, no star trader wants to work for a firm that can't handle disaster.