At 15:52 on Friday 22nd April 2016, several customers reported agents being logged out, screens freezing and/or calls not completing. The MSO process was started, but symptoms had cleared by this time. A critical software push was deployed at midday on the Easy Contact Now platform. This was successful, but a part of that process made a bulk change to database later in the afternoon which slowed the system for the duration of the transactions.

Timeline

15:52: Certain transactions running slowly

15:57: Backlog of transactions clears

16:03:MSO process begun

16:05: Engineers confirm system had already automatically recovered

16:20: Root cause identified.

Root Cause

A hot-fix software push created a long running database query which then slowed platform performance.

Preventative Actions

The change management process was changed such that hotfixes are only pushed at night.