At approximatelyÂ 2:12 PM CDT our systems detected a network event in our St. Louis facility. At this time the event has cleared and our engineering team has completed their investigation. The RFO for this event has been published to our kb and can be found at /bit.ly/2NJapBe

If you have any further questions or concerns, please contact your support team.

Scope of Work:Â On Monday, June 25th, 2018 Contegix will be performing the quarterly battery inspections for all UPS units located at our 210 North Tucker 6th Floor datacenter. This maintenance is not expected to impact customers and is performed as a preventative measure to ensure our uninterruptible power supply continues to perform as expected. Â As maintenance will be conducted on 1 UPS unit at a time, this maintenance will not affect the normal daily operations of our 210 North Tucker 6th Floor datacenter.

Scope of Work:Â The Contegix Networking Team will be performing maintenance on the Contegix St. Louis locationâs internal management network.

Impact on Customers:Â Contegix customers should use the current Contegix Portal (swift fox) and support phone lines (877) 289-0395 ext. 2 for access to support while the maintenance is ongoing.Â During this maintenance, the legacy Contegix Portal (trierarchal) will not be accessible. Disaster recover replication tasks from STL to other facilities will be delayed.

Contegix is currently investigating alerts and reports of packet loss from the St. Louis location.Â Networking engineers are evaluating the problem and updates will follow.

Update:Â Contegix Networking has isolated and resolved an issue believed to be the source of the problem.Â Further investigation is on going.Â Please contact the Support team if you are still experiencing packet loss or performance issues.

Scope of Work:Â The Contegix Networking Team will be performing maintenance on the Contegix St. Louis locationâs internal management network.

Impact on Customers:Â Contegix customers should use the current Contegix Portal ((310) 416-5171) and support phone lines (877) 289-0395 ext. 2 for access to support while the maintenance is ongoing.Â During this maintenance, the legacy Contegix Portal (580-816-3598) will not be accessible. Disaster recover replication tasks from STL to other facilities will be delayed.

Scope of Work:Â The Contegix Networking Team will be performing maintenance on the Contegix St. Louis locationâs internal management network.

Impact on Customers:Â Contegix customers should use the current Contegix Portal (/contegix.service-now.com/) and support phone lines (877) 289-0395 ext. 2 for access to support while the maintenance is ongoing.Â During this maintenance, the legacy Contegix Portal (/portal.contegix.com/) will not be accessible. Disaster recover replication tasks from STL to other facilities will be delayed.

Contegix is currently investigating a potential issue with some virtual machines hosted on the St. Louis based Classic Cloud environment.Â Additional updates will be provided as they becomes available.

Update (2:04PM CDT): Contegix Engineering has isolated the issue with the Classic Cloud environment to a storage system issue and is currently working to resolve issues.

Update (2:12PM CDT): The Classic Cloud platform has stabilized and Contegix Support is currently working on resolving issues on individual virtual machines.Â Â If you continue to have issues, please contact the Support team via our portal.

At approximately 6:40 AM CDT our engineers began to observe communication issues which affected some systems within our St. Louis datacenter. This issue was immediately escalated for investigation.Â We are currently working to resolve this issue and investigating the root cause. Additional updates will be provided as they becomes available.

Update: Our Engineering team has identified the root cause of the outage as a problem on our shared storage platform.
We have escalated this issue and are currently working with our storage vendor for a resolution.

Update:Â Our Engineering teams continue to work with the vendor on complete resolution, but at this time we are able to bring some customers back online manually.Â We will continue to update as we are able to bring more systems online.

Update: 10:48AM CDT – We continue to bring environments back online and expect to have all systems back up within the next 2 hours. Some VMs are going to require actions to be taken at the operating system level and our teams will continue to work through those on a customer by customer basis.

Update: 11:41AM CDT – Our engineering team has continued to bring individual environments back online and we estimate to have all systems back online within 60 minutes.Â There are still going to be actions that need to be taken to recover VMs.Â If you’re still experiencing issues and don’t currently have a support case open, please open a support ticket with the details of the issue at our support portal.

Update: 12:43PM CDT – At this time, the platform has been stabilized. Our teams are still working through individual system failures to bring all systems back online.Â Please open an incident with your support team, if you are still experiencing impact and do not have one open already.

Update: 2:38PM CDT – The platform has been stabilized and weâre continuing to work through individual host based issues.

Update 3:34PM CDT – With the platform stabilized, this will be the final global update for this incident. If you continue to have impact, please contact the Support team via our portal.