This is a political forum that is non-biased/non-partisan and treats every persons position on topics equally. This debate forum is not aligned to any political party. In today's politics, many ideas are split between and even within all the political parties. Often we find ourselves agreeing on one platform but some topics break our mold. We are here to discuss them in a civil political debate. If this is your first visit to our political forums, be sure to check out the FAQ and RULES. Registering for debate politics is necessary before posting. Register today to participate and remove the ads - it's free!

Re: Trading halted on NYSE floor

What do you think United's operational uptime percentage is? The NYSE? Why would the NYSE engage in an upgrade mid week? Why did that anonymous tweet seem to be in the know?

yeah it was only the dog and pony trading floor, but its far too coincidental to believe the reports.

If my system went down for 3 hours, people would be fired and providers would be sued. we operate above that %99.999 threshold. do you think the NYSE or united, or the WSJ would operate any less efficient?

Emergency release to fix something else that it was decided couldn't wait until the weekend. I don't know that that's what happened but it's an often enough occurrence.
Or how about an upgrade over the weekend that didn't actually exercise the problem code path until yesterday?

How do you define your 99.999% uptime? I ask because it's impossible to make it meaningful in the real world for any moderately complex system.

Re: Trading halted on NYSE floor

Originally Posted by ReverendHellh0und

wait this is a hoot....

"NYSE and NYSE MKT began the process of canceling all open orders, working with customers to reconcile orders and trades, restarting all customer gateways and failing over to back-up trading units located in our Mahwah, NJ datacenter so trading could be resumed in a normal state"

Mahwah is the backup now? (it's not, SHHHHH)

and it took them 3 ****ing hours to fail over?

Many of my customers have fault tolerance and redundancy, "failover" ranges from near instant, to about 10 mins.

If it's a software problem all the hardware in the world won't help you.

Re: Trading halted on NYSE floor

Originally Posted by ReverendHellh0und

If my system went down for 3 hours, people would be fired and providers would be sued. we operate above that %99.999 threshold.

That may well be true, but without knowing what sort of system you manage, I can't say whether it has the kind of complexity, traffic and the very narrow fault tolerances something like the NYSE does. If one trade in a million fails to complete for a technical reason, that's a huge, show-stopping, problem for the NYSE. Even if certain transactions just take 100 milliseconds longer to complete than they should, that is like alarms start going off an people panic. They can't rely on the usual techniques of letting things fail over after things time out or using not-quite-synchronous data and whatnot.

Also, the system you manage is probably more similar to a lot of other systems, so you have to deal with bugs in the slice of the environment that is custom built code for that particular purpose, but most of the code you're using is like Apache, Linux, Windows, Oracle, etc.- things that have been tested and fixed up for many years and which serve millions of users. In the NYSE, the odds are nearly all their code is custom. Even most their hardware and network protocol is custom.

So, I have no idea. It could have been Anonymous for all I know. But I wouldn't go around comparing uptime you see in a typical IT situation to uptime at something like the NYSE. A lot of what you would count as "up" would count as "down" for the NYSE and they're working in a much more complex, bespoke, environment.

Re: Trading halted on NYSE floor

Neither of these small glitches seemed to have involved mid week "upgrades" and forgetting to make sure the clients were compatible. while yes of course they may at times have issues, it does happen, the "we did an upgrade" doesn't make much sense.

You have a system that 20,0000 people connect to and has 15 functions. If one person cannot execute one function but everything else works is the system up or down? How does that failure get factored in uptime calculations?

I'll refrain from commenting on anything specific to NYSE since I work for them - though not in the area affected yesterday. I will say we don't as general rule make routine production changes during the week. That doesn't mean we don't ever make them should circumstances warrant. It also doesn't mean anything that the release didn't use the words "emergency"

Re: Trading halted on NYSE floor

Originally Posted by tuhaybey

That may well be true, but without knowing what sort of system you manage, I can't say whether it has the kind of complexity, traffic and the very narrow fault tolerances something like the NYSE does. If one trade in a million fails to complete for a technical reason, that's a huge, show-stopping, problem for the NYSE. Even if certain transactions just take 100 milliseconds longer to complete than they should, that is like alarms start going off an people panic. They can't rely on the usual techniques of letting things fail over after things time out or using not-quite-synchronous data and whatnot.

We have clients who have similar requirements and have HA in place. we also don't do service upgrades in the middle of thier busy time.

Also, the system you manage is probably more similar to a lot of other systems, so you have to deal with bugs in the slice of the environment that is custom built code for that particular purpose, but most of the code you're using is like Apache, Linux, Windows, Oracle, etc.- things that have been tested and fixed up for many years and which serve millions of users. In the NYSE, the odds are nearly all their code is custom. Even most their hardware and network protocol is custom.

WE have both, and we manage several system, not jut one from our noc.

So, I have no idea. It could have been Anonymous for all I know. But I wouldn't go around comparing uptime you see in a typical IT situation to uptime at something like the NYSE. A lot of what you would count as "up" would count as "down" for the NYSE and they're working in a much more complex, bespoke, environment.

are you suggesting the stock exchanges networks are more apt to go down than say, United? oh wait. O

Re: Trading halted on NYSE floor

We have clients who have similar requirements and have HA in place. we also don't do service upgrades in the middle of thier busy time.

WE have both, and we manage several system, not jut one from our noc.

are you suggesting the stock exchanges networks are more apt to go down than say, United? oh wait. O

I mean, I don't know what kinds of systems you manage, so I can't really say, but here, take an example. Say that in one of the applications you manage, somebody submits a helpdesk ticket saying that their user profile page isn't loading. How long might it be before anybody checks into the ticket at all? If the page in fact is not loading, how long before somebody diagnoses the problem? How long before you get a fix on the system typically?

Now, if instead of a user's page not loading, that was a situation where somebody's trade wasn't being recorded, that could leave the NYSE user in a situation where they lose $600,000 when the stock goes up 1% before they realize that the trade wasn't actually recorded. A similar severity of bug as might cause the user above's profile page not to load might cause a trade not to be recorded, but where in the profile page scenario, you consider the time that bug is in place to be uptime, and the bug may be in place for weeks, months or even years, the NYSE would immediately shut down and would not come back up until they found and fixed the bug and corrected any data that had been corrupted by it.

Generally speaking, places that shoot for 99.999% uptime are places where users can deal with a few bugs and where the system behaving slowly for brief periods still counts as it being up. There are not a lot of environments that I'm aware of with 99.999% uptime where "up" means zero bugs and nearly instantaneous response times. And I can't think of any that hit that kind of zero bugs, instantaneous response, uptimes with a massive worldwide distribution of networks, servers and software doing extremely complex things running all custom software.

For example, Google's search engine is massive and equally complex and it hits 99.999% uptime. But, it does it by being very fault tolerant. If you do the same search 100 times, you won't actually get quite the same results each time because different databases will be getting fed new data at different times and this thing won't be quite up to date on that one, but will on this other, etc. Many of the stats they give you, for example about search queries, the data you get back is in the form of "approximately 500 - 540" and whatnot because they're often using more scalable, faster, techniques when they don't need to be 100% confident in 100% accuracy. Other large complex systems often hit 99.999% by being slow. For example, most ecommerce platforms have no problem making a user wait even 5 or 10 seconds while they process things. The NYSE can't do either of those things, and I can't think of another equally complex system that can't do either of those things which hits 99.999% with no bugs.