Only 7 minutes later has our SRE (Site Reliability Engineering) team been notified of the issue and initiated immediately our alert chain procedure. A communication has been published on our public help Center Web site at 08:00.

A dedicated communication to our customer administrators has been sent by email at 11:01

While this unprecedented outbreak was completely out of our control, we owe our customers our deepest apologies.

Rainbow services have been restored in a degraded mode by 4:45pm and the complete infrastructure has been restored to its original state by 11pm. The root cause was a default in the data-center power supply entry point, causing both redundancy points to be shutdown simultaneously.

In the immediate aftermath of power restoration, many core routers required reconfiguration. Some of the 40k servers, including Rainbow ones needed to be replaced due to power-damaged components, leading to hours of struggle.

While we thought having taken all necessary means on Rainbow operations side to ensure a full scale redundancy of all of our services, experience has proven us wrong. We need to be better prepared and we will be.

While all application services are designed to be highly available, the data-center itself was our single point of failure. Our roadmap included to add geographical redundancy to our data centers for early 2018. This plan has been accelerated to be ready before end of 2017.

As of 2018, our current core infrastructure will be hosted on 3 geographically isolated sites in France and same principles will be applied when opening new countries and continents.

In addition, communication to our Rainbow administrators will be improved to decrease timeframe between the outage and the dedicated communication.

HISTORY:

We are currently experiencing a major outage on Rainbow services. While all means have been taken on Rainbow operations side to ensure a full scale redundancy of all of our services, the underlying power supply and networking backbone of our IaaS partner has been compromised. We are actively working with our partner to restore Rainbow services as soon as humanly possible and will communicate further whenever progress are made.

Friday 10/11/2017 - 2:30 PM CET

Rainbow Services are now fully restored. The last maintenance window was done at 12:30 PM CET but you probably was not impacted. A detailed "Lesson Learned" is coming soon.

Friday 10/11/2017 - 9:15 AM CET

Our team worked during the night to restore full services. You have maybe noticed some troubles between 3:00 to 4:30 AM CET & 6:45 to 7:30 AM CET. Service is back online, nevertheless you can face issues with PBX connectivity. A new maintenance window is planned during the day.

Desktop application:Due to the time of unavailability, it is recommended you stop and start rainbow to successfully reconnect. A computer reboot will also clear any connection issue.

Thursday 09/11/2017 - 6:15 PM CET

Services are restored and Rainbow is back. Nevertheless, some services do not work properly and are functioning with diminished capacities. We are still working on it.

Thursday 09/11/2017 - 4:30 PM CET

Rainbow infrastructure has been partially restored. We still have issues to access to some servers. Our Cloud Provider is investigating these issues.

Thursday 09/11/2017 - 2:15 PM CET

Our Cloud Provider solved some public access issues but he's still working on other blocking points. We are working on Rainbow service restoration in parallel to have Rainbow up as soon as possible.

Thursday 09/11/2017 - 1:30 PM CET

Our Cloud Provider needs to restart some services. Rainbow services check will be launch as soon as our administration access is available again.

Thursday 09/11/2017 - 11:00 AM CET

Routing services restoration is under progress. Our Cloud Provider expects full network availability in 1 to 2 hour for infrastructure. Rainbow services check will be launch as soon as the infrastructure is available.

Thursday 09/11/2017 - 10:30 AM CET

Our Cloud Provider restarted their Datacenter power supply. Network services should be up again in about 30 minutes. Once the connexion is available we will perform some checks about Rainbow services and provide you further details.

Thursday 09/11/2017 - 9:30 AM CET

Our Cloud Provider partially restarted their Datacenter power supply. We are waiting for more details about network service restart.

Thursday 09/11/2017 - 9:00 AM CET

Our Cloud Provider is working with French network electricity distributor manager to solve power supply for the Datacenter.

Thursday 09/11/2017 - 8:00 AM CET

We are experiencing an issue with our cloud provider. All rainbow services are down.Investigations are ongoing with our Cloud Provider support.