Posts Tagged 'Incidents'

Since posting Improving Communications for Customer-Affecting Events, our developers have been hard at work making the SoftLayer customer portal even better. Initially, as discussed in the previous post, we released functionality that allows us to more rapidly communicate with customers about unplanned events such as infrastructure troubles or outages. The tool also allows customers to read and follow updates. Communications are targeted to specific customers who may experience degradation of disruption to their services as a result of these events.

I’m pleased to report a new major milestone: The system is now ready to distribute planned events communications such as scheduled maintenance.

Until now, SoftLayer used read-only tickets as the method to post targeted communications to customers about upcoming planned events. Here are some of the customer benefits from moving to the Event Management System (planned events; unplanned events) within the customer portal:

With the mass ticket creation, only the master user account received email alerts. Now, any portal user account with a valid email address and permission to manage bare metal servers or virtual servers are now subscribed to receive the new planned event notifications.

Events are clearly organized in their own area in the portal—separate from tickets. The overview list shows relevant columns such as planned start and end times.

Customers only receive notifications relevant to their infrastructure.

The portal and email notifications will include a table listing which items on your account may be affected.

An alert bar will appear at the top of the customer portal if there is an active event in progress, which helps customers quickly find relevant information about any service impact.

A notification will be shown in the customer portal if customers open a ticket about a bare metal server or virtual server that is associated with an event in progress.

SoftLayer operational staff can rapidly post important updates to events as they arise.

No additional customer action is required in order to benefit from these improvements. However, you may wish to familiarize yourself with the Events view, which can be found under Support > Events.

When we publish information about a planned event that may affect your service, it will appear highlighted on the calendar. Customers can click directly on the date in the calendar to see an event planned for that day.

As mentioned previously, all users are subscribed by default to receive planned event email notifications. You may opt out of these notifications in the Account > Subscriptions dropdown menu.

We’ve already begun to use the new Event Management System for planned events as of December 2014. By the end of January 2015, we will cease using the system which opens up read-only tickets and exclusively use the Event Management System for future planned events.

Please take some time to familiarize yourself with this area of the customer portal, ensure that you have a valid email address associated with your login, and check your subscription settings for planned and unplanned events. Additionally please note that the Master User, as well as accounts that administer customer logins, can control subscription settings for their respective sub-users.

During the next phase of improvements to customer communications, our developers will be focusing on implementing these events in the SoftLayer mobile (smartphone/tablet) client.

Service disruptions are never a good thing. Though SoftLayer invests extensively in design, equipment, and personnel training to reduce the risk of disruptions to our customers, in the technology world there are times where scheduled events or unplanned incidents are inevitable. During those times, we understand that restoring service is top priority, and almost as important is communicating to customers regarding the cause of the incident and the current status of our work to resolve it.

To date we've used a combination of tickets, emails, forum posts, portal "yellow" notifications, as well as RSS and Twitter feeds to provide status updates during service-affecting events. Many of these methods require customers to "come and get it," so we've been working on a more targeted, proactive approach to disseminating information.

I'm excited to report that our Development and Operations teams have collaborated on new functionality in the SoftLayer portal that will improve the way we share information with customers about unplanned infrastructure troubles or upcoming planned maintenances. With our new Event Communications toolset, we're able to pinpoint the accounts affected by an event and update users who opt-in to receive notifications about how these events may impact their services.

As the development work is finalized, we plan to roll out a few phases of improvements. The first phase of implementation, which is ready today, enables email alerts for unplanned incidents, and any portal user account can opt-in to receive them. These emails provide details about the impact and current status of an unplanned incident in progress (UIP). In this phase, notifications can be sent for devices such as physical servers, CCIs and shared SLB VIPs, and we will be adding additional services over time.

In future phases of this project, we plan to include:

A new "Event" section of the Customer Portal which will allow customers to browse upcoming scheduled maintenances or current/recent unplanned incidents which may impact their services. In the past, we generated tickets for scheduled maintenances, so separating these event notifications will improve customer visibility.

Enhanced visibility for events in our mobile apps (phone/tablet).

Updates to affected services for a given event as customers add / change services.

Notification of newly added or newly updated events that have not been read by the user (similar email "inbox" functionality) in the portal.

Identification of any related current or recent events as a customer begins to open a ticket in the portal.

Reminders of upcoming scheduled maintenances along with progress updates to the event notification throughout the maintenance in some cases.

Improved ability to correlate specific incidents to customer service troubles.

Dissemination of RFO (reason-for-outage) statements to customers following a post-incident review of an unplanned service disruption.

Since we respect our customers' inboxes, these notifications will only be sent to user accounts that have opted in. If you'd like to receive them, simply log into the Customer Portal and navigate to "Notification Subscriptions" under the "Administration" menu (direct link). From that page, individual users can control event subscriptions, and portal logins that have administrative control over multiple users on the account can control the opt-in for themselves and their downstream users. For a more detailed walkthrough of the opt-in process, visit the KnowledgeLayer: "Update Subscription Settings for the Event Management System"

The Network Operations Center has already begun using this customer notification toolset for customer-affecting events, so we recommend that you opt-in as soon as possible to benefit from this new functionality.