Archive

Hurricane Sandy Related Updates

Written by Dan Hoffman | October 30, 2012

As a result of massive flooding and power loss caused by hurricane Sandy, ShoreTel Sky began to lose some services in two of our data centers at about 10:45 pm on October 29. Multiple carriers and our data center providers are reporting issues. We will update this page as information becomes available and continue to update as additional details emerge.

Daily Update: 11/1

11/1/12 5:00 PM EST:

Dear ShoreTel Sky Customers,

We are happy to report that at this hour all of our core systems are stable. Some clients continue to be impacted by 3rd party carriers, particularly Version and Broadview, who are experiencing storm related service interruptions. We will continue to work with them in any way we can and may yet re-route DS3/T1 traffic to another carrier.

The remaining issue with conferencing on one of our clusters has been resolved. We are experiencing some difficulty with ShoreTel Sky help. Engineers are working on it, but in the meantime, rather than pressing the “?” button for help, please email us or call our support hotline -1-877-654-3573. We have also posted some very specific tips on working remotely that you might find helpful.

At this point, we are going to resume our normal method of communicating system status with clients through our Trust website. This will be the last systems update on this blog. Certainly if the situation changes and normal communication is not sufficient we will resume these updates.

On Wednesday, November 7 we will hold an executive briefing for customers to discuss the impact of storm Sandy along with other service updates. Clients will also have the opportunity to ask questions of our team. Watch your email for an invitation or ask your Account Manager for details.

We’d like to thank all of our customers for their patience and understanding through this event. We hope you are all safe and will be able to resume normal business operations soon if you have been challenged by events in the North East. As always, don’t hesitate to reach out if there is anything we can do to help.

1:30 PM EST: While all of our core voice systems remain operational, we continue to work with some 3rd party carriers who are experiencing service interruptions due to major flooding. Most significantly, Verizon, the incumbent data carrier in NYC, is still without power at their downtown central office. As a result, other carriers who rely on Verizon are impacted. We are actively researching the possibility of routing DS3/T1 traffic to another carrier if possible.

Other Notes:

We have reports of conferencing issues on one specific cluster (NED). Our engineers are actively working on this issue.

We encourage clients to make sure that their e911 settings match their current location. This can be done in the ShoreTel Sky portal

We have some reports of call quality issues for calls to our support toll free numbers. If you experience trouble reaching us, please feel free to call us at 1-877-654-3573

11/1/2012 - 9:30 AM EST: Some of the major carriers with Central offices in Manhattan are reporting service disruptions due to flooding. We’ve been advised they will begin installing new equipment to replace what was damaged by the flooding. Currently they are not able to provide an ETR.

As a result of the situation reported by our carriers, a subset of our "on-net" clients may experience dropped calls and/or loss of internet connectivity while switching automatically to a backup connection. Clients without backup connectivity will likely lose voice and data service during such an incident. However, inbound calling to applications such as voicemail and auto-attendants should not be affected by this type of issue.

If you notice this issue please refer to our blog post "After The Storm" for helpful tips on how to continue operations

As always, if you are experiencing a particular difficulty, please reach out to our support team.

1:30 PM EST: While all of our core voice systems remain operational, we continue to work with some 3rd party carriers who are experiencing service interruptions due to major flooding. Most significantly, Verizon, the incumbent data carrier in NYC, is still without power at their downtown central office. As a result, other carriers who rely on Verizon are impacted. We are actively researching the possibility of routing traffic to another carrier if possible.

Other Notes:
We have reports of conferencing issues on one specific cluster (NED). Our engineers are actively working on this issue.

We encourage clients to make sure that their e911 settings match their current location. This can be done in the ShoreTel Sky portal

We have some reports of call quality issues for calls to our support 800 numbers. If you experience trouble reaching us, please feel free to call us at 1-877-654-3573

11/1/2012 - 7:00 AM EST: We have been in contact with both collocations and have been assured they are in good shape with fuel levels and are currently stable. We will continue to stay in close contact with them as conditions are subject to change.

We are actively investigating isolated issues of ShoreTel Sky Replay.

Now that our core systems are stable, we will be moving back to utilizing our standard maintenance windows to perform work on items requiring system updates to mitigate client impact during normal business hours.

Our focus working closely with our vendors will continue to ensure connectivity is stable going forward.

With the situation now stabilized, we will post updates to this blog only when we have material information to share.

As always, if you are experiencing a particular difficulty, please reach out to our support team.

Daily Update: 10/31

10/31/2012 - 7:00 PM EST: Our core systems remain stable. We are working on isolated issues related to ShoreTel Sky RePlay and ShoreTel Sky Fax that have been reported. Our Quality Assurance team is reviewing the final tests for ShoreTel Sky Connect and ShoreTel Mobility.

Now that the situation has stabilized, we will post updates to this blog only when we have material information to share. Rest assured that our teams will continue to work tirelessly to ensure the continued operations of our systems and to support our clients. Should the situation change, we will resume our hourly updates.

As always, if you are experiencing a particular difficulty, please reach out to our support team.

We will provide another update at 7:00PM EST to confirm stability, then we will resume regular updates in the morning unless a change in conditions warrants more frequent updates through the night.

10/31/2012 - 5:00 PM EST: We have received feedback from some clients that the one-way audio issues remain an issue. Our engineering teams are focusing on this issue.

We will continue to provide hourly updates as we work to resolve this issue. If you are experiencing any other issues, please report them to our support team.

10/31/2012 - 3:00 PM EST: We would like to reconfirm that the following systems have been restored with power via generators: all soft switches, ShoreTel Sky Contact Center, ShoreTel Sky Portal, ShoreTel Sky Communicator, trust.shoretelsky.com and our support website.

We are troubleshooting sporadic reports of one-way audio with certain carriers as well as one particular carrier that has an inbound one way talk path.

In addition, our vendor who had 800 number problems has also reported that the problem is now resolved.

While we have made significant progress in restoring our core systems and are rapidly approaching a point where all of our core systems are restored, we are not out of the woods yet.

First, similar to many other carriers, we are still running on generator power in all of our New York City based datacenter collocation facilities. At this time, all of our providers are reporting that their situations are stable, but they cannot yet guarantee against future power interruptions given the situation in the city. Note that the City is also cautioning businesses to expect many days without power in Manhattan. We are reviewing our staffing and procedures to make sure we are on the front-foot and ready in the event of a generator issue at any center.

Second, there are still widespread issues in both third-party telephone networks and bandwidth networks. This impacts our vendors directly, as well as the vendors of those you interact with over the phone. This means that while our systems are running, customers may still notice difficulty in completing some calls. These could be issues of our carrier or the carrier of the party you are speaking with - we will need your help to identify where it is occurring.

We have been in contact with our vendors throughout the crisis to understand their issues caused by the storm. As we work through the remaining core issues, our technical teams will turn to helping our vendors isolate upstream issues which are negatively impacting the ShoreTel Sky systems and customers.

The most pressing of these issues we are investigating is a one way talk path issue on inbound calls with a particular vendor. We are aggressively escalating this issue.

It is helpful to us to hear from you about specific issues and we continue to staff our support lines. In addition, we may be able to help you use the capabilities of the system to maintain connection to your staff working from outside offices, and getting your customers through to the right people quickly. Please contact our support team if you would like this assistance.

Our next update will be at 3:00 PM EST.

10/31/2012 - 12:30 PM EST: ShoreTel Sky Contact Center has been validated and is working properly. We have found that if the carrier is able to deliver calls to our system, we route the calls to agents and they are able to answer them. Agents are able to log in, go ready and answer calls. Recordings and dashboards are working as expected.

Upstream carriers are going on and offline due to their issues caused by hurricane Sandy, which is negatively impacting call completion as well as call quality. Although, many of these issues are outside of our control, we are working with our carriers and peers toward resolution.

Engineering continues work on the TAU and SKY soft switch issues after last night’s power failures. We have isolated the remaining issues and are working to correct them asap.

The vendor that provides most of our 800 number service is reporting a higher level outage with the SS7 network, due to their post-hurricane issues. Their technical team is working on it and they have escalated with other carriers to isolate the issue. We are still available directly at1-877-654-3573.

10/31/2012 - 11:15 AM EST: The remaining soft switches requiring work, TAU and SKY, continue to beaddressed by engineering. Other systems including ShoreTel Sky Portals are coming back online soon. We will confirm when access has been restored.

ShoreTel Sky Contact Center is currently online but being tested by our engineers. Customers may wish to test it, although we have not yet validated that everything is working properly.

10/31/2012 - 10:00 AM EST: The remaining soft switches requiring work, NED, TAU and SKY are being addressed by engineering. Other systems including ShoreTel Sky Portals and ShoreTel Sky Contact Center are also being restored and coming back online. We will confirm when access has been restored.

We have received reports from our partner carrier of inbound calling failures as well as circuit failures. We are addressing these issues with our partners, however many of them rely on outside parties that are also having difficulties, in particular, one of the 800 number carriers who is experiencing an outage.

Clients who are calling ShoreTel Sky support from a ShoreTel Sky phone should dial: 9 – 1-877-654-3573. All other callers should dial 1-877-654-3573.

10/31/2012 - 9:00 AM EST: The remaining soft switches requiring work, NED, TAU and SKY are being addressed by engineering. They are experiencing complications related to the power issue experienced at one of the collocation facilities overnight. Engineers are actively working on restoring service.

10/31/2012 - 8:00 AM EST: Our two major collocation facilities in New York City have restored power. Engineering teams are working in both collocation facilities to validate that services are coming back up as expected.

We will provide the next update at 9:00 AM EST.

10/31/2012 - 7:00 AM EST: The collocation facility reports progress to restore power soon. We do not yet have an ETA from them. Our daytime staff are fully on board and efforts continue to restore services.

We will provide the next update at 8:00 AM EST.

10/31/2012 - 6:00 AM EST: Our overnight team continues to make progress, with focused efforts on full resolution of all remaining issues and restoration of all remaining services. However, we have just received word from the collocation facility that the facility is offline again due to their power issues. They are working with building personnel to rectify asap

We will provide the next update at 7:00 AM EST.

10/31/2012 - 3:00 AM EST: Data center engineers have confirmed that the generator is running normally. They also report that they believe that ample fuel is on site and additional fuel deliveries are being coordinated. Progress on site infrastructure has been made but the data center is still not permitting client access.

Our overnight team continues to work and has completed system checks on all clusters to pinpoint remaining issues and continues focused efforts on resolution.

We will provide the next update at 6:00 AM EST.

10/31/2012 - 12:00 AM EST: The remaining impacted data center has reported that power to the facility has been restored. Data center operators are not allowing any access to the site yet, but will let us know as soon as access can be permitted. Our engineers continue their work, but will need access to the data center to address the remaining service issues.

We will provide the next update at 3:00 AM EST.

Daily Update: 10/30

9:00 PM EST: Updates from data center operators regarding power resumption are delayed, but are expected soon. Our engineers will continue their work throughout the night to restore remaining affected services.

We will provide the next update at 12:00 AM EST

8:00 PM EST: We expect an update from the operator of the impacted data center within the hour. No other new updates are available.

We will provide the next update at 9:00 PM EST. After that, we will provide updates to the blog at 12 AM EST, 3 AM EST, and 6 AM EST. Then we will resume our hourly rhythm of updates on the blog and via email until all services are restored and stable.

7:00 PM EST: No new information regarding power resumption for customers still impacted is available.

Our engineers are working to restore remaining affected services such as the ShoreTel Sky Contact Center and the ShoreTel Sky Portal.

We will provide updates at 8:00 and 9:00 PM EST. After that, we will provide updates to the blog at 12 AM EST, 3 AM EST, and 6 AM EST. Then we will resume our hourly rhythm of updates on the blog and via email until all services are restored and stable.

6:00 PM EST: At this time, we do not have additional feedback as to when the power for our remaining clients affected by this outage will be restored. We are expecting another update from the data center provider very soon and will provide it in our 7 PM EST update.

We will continue hourly updates from now until 9 PM EST. After that, we will provide updates to the blog at 12 AM EST, 3 AM EST, and 6 AM EST. Then we will resume our hourly rhythm of updates on the blog and via email until all services are restored and stable for all clients.

5:00 PM EST: The biggest issue is still the data center that lost power. The good news is that their fuel truck is now on site at the data center.

Their current issue is related to refueling the storage tanks that feed the generators on the mezzanine level of the data center. There are 3 teams involved in this effort.

If all goes well they will have fueled generators generating power later tonight. Worst case scenario, noon tomorrow. Once the power is restored, we may have some work to do to fully restore services.

We expect another update from the data center at approximately 6 PM EST.

4:00 PM EST: We have made progress with our efforts to bring up portal.shoretelsky.com. We are hopeful that web service will be restored shortly.

At this time, a data center is still without power. The data center’s provider has coordinated fuel deliveries and pumps, as well as engineers to fabricate a pipe to bring the fuel directly to the generators.

There is currently no ETA available for the completion of these activities, but the provider is working through all options to bring the UPS infrastructure back online as soon as possible. We will continue to work toward resolution and provide status updates with any new information.

Thank you for your continued patience and understanding during this crisis.

3:00 PM EST: Our efforts to restore service continue. The status has not changed much since our last update.

Ninety five percent of service at the major data center in NYC has been restored. We continue to work on one system that house approximately 5% of our service in that location. Beyond that system, we are working to restore web applications that accompany the phone service. In the other data center, which houses approximately 33% of our clients, we continue to communicate with our data center provider who is having power issues.

Beyond these facilities, many other carriers continue to have incidents that may be impacting call completion, http://reut.rs/Udo3W9. We remain focused on bringing up our own systems first.

We are taking steps to forward all phone numbers that are out of service to a message so that your organization's callers receive a message rather than a fast busy signal. They will receive the following message:

"Thank you for your call. Due to Hurricane Sandy, our communication systems are currently out of service. Please try your call again later." The call will then disconnect.

We will provide another update on the hour.

2:00 PM EST: Our efforts to restore service continue. The status has not changed dramatically since our last update.

Ninety-five percent of service at the major data center in NYC has been restored. We continue to work on one system that houses approximately 5% of our services in that location. Otherwise, we are working to restore the web applications that accompany the phone service. Power issues persist in a second data center, which houses approximately 33% of our clients. We are in constant communication with our partner provider.

Beyond these facilities, many other carriers continue to have incidents that may impact call completion, http://reut.rs/Udo3W9. We remain focused on bring up our own systems. We will provide another update on the hour.

1:00 PM EST: Our efforts to restore service continue. Of the two data centers most heavily impacted in NYC, 95% of service at our major data center in NYC has been restored. Our other data center housing approximately 33% of our clients is experiencing problems, having depleted its fuel reserves for generators.

That said, beyond the issues that our data center providers are experiencing, we are hearing from our carriers that there are wide-spread outages that impact call completion before they reach our network. Verizon, the incumbent carrier in Manhattan is reporting major damage to facilities serving all carriers, http://reut.rs/Udo3W9.

One of our major partner carriers is reporting 7 incidents opened with other carriers including Verizon, AT&T, XO Communications, and Frontier to name a few. On the access side, our partner carrier has over 40 Incidents opened for their services from Allentown, PA, through Manhattan to parts of New Jersey.

12:00 PM EST: Most of Manhattan, 90% of Long Island, and much of NJ are without power. Many of our carriers are reporting a "state of emergency." A few minutes ago, one of the major data centers in NY ran out of diesel power. City safety restrictions are hampering fuel delivery. This impacted a number of our clusters that had previously been restored. We will provide updates as soon as we know more.

Clients who are calling ShoreTel Sky support from a ShoreTel Sky phone should dial: 9 – 1-877-654-3573.

11:00 AM EST : Most core voice services have now been restored. We continue to work actively to restore all services in conjunction with our partners and service providers. Based on news from our datacenter providers, there remains some risk of service interruption throughout the day, given the unique and difficult conditions. We will continue to work actively though out the day to remedy the situation and keep our clients informed.

10:06 AM EST: As you know, Hurricane Sandy caused unprecedented flooding and power loss throughout the greater New York area. Unfortunately, along with many other major carriers, two of our data centers have been impacted by a combination of failures to building back-up power, and the city taking unprecedented actions in terms of evacuation. We have been alerted to expect issues to continue throughout the day. While many services in Chicago are working, there are some dependencies on NY carriers and equipment that have impacted ShoreTel Sky clients across North America Our engineers have been working throughout the night and have restored many services. We will continue to do work actively throughout the day. Our support lines continue to be open.

We will continue to post updates on an hourly basis (at the top of each hour) to www.ShoreTel.com and www.shoretelsky.com. Once services have been fully restored, we will evaluate our response to the situation and will provide a more thorough explanation of what happened .

7:40 AM EST : Our 877 654 3573 support number has been working throughout the night. ShoreTel Sky staff are available to assist you. However, please only call with emergencies. Thank you.

7:00 AM EST : Some services have been restored as of 6 am. However, there continue to be widespread failures of primary and backup power at major NYC central offices and telecom hubs. We just received notice that flooding at one of the data centers is preventing fuel from reaching their backup generators, and some clusters may lose service again shortly. The ShoreTel Sky team continues to work throughout the night. Please return here for updates.