The primary SmarterMail server (MX01) uses dual power supplies for redundancy in case of a single PSU failure. After contacting Supermicro we were notified that the internal power distribution unit that the dual power supplies plug into had failed causing the server to receive no power. This is not a component that is stocked at our data-center and not something that can be picked up locally and must be ordered directly from Supermicro.

The server is currently operational, but using a standard ATX power supply. Once we’ve received the replacement part from Supermicro, maintenance will be scheduled to replace it so that the server can be restored to full working order.

We’d like to thank everyone for their patience and understanding during this time and for choosing ASPnix!

We are currently investigating an outage of our primary SmarterMail server. At this time the server is unavailable. Incoming email will be captured by our backup gateways and will be delivered once the primary server resumes operations.

We are very pleased to announce support for the new .NET Core 1.0! All web servers have been updated with the ASP.NET Core 1.0 runtime as well as the IIS modules, customers can start deploying ASP.NET Core web applications immediately!

The new runtime improves performance by focusing and optimizing your web application to run on the web server and the cloud environment. As well as the ability to run your application across platforms such as Windows, Linux and macOS.

.NET Core is available for those who’d like to start developing or deploying ASP.NET Core applications right now, simply select “No Managed Code” in the control panel for your website’s ASP.NET version and then publish your application.

We are proud to support .NET versions from 2.0 all the way to 4.6.1, including the new .NET Core 1.0.

There will be a brief interruption of our Windows Control Panel system around 11:30AM MDT for about 30 – 60 minutes while we complete the migration of the system to a new server and complete some additional upgrades.

During this time customers will not be able to access the Windows Control Panel and scheduled tasks will not be executing. Once the migration and upgrades are completed the system will resume normal operations. This maintenance will improve the performance of the control panel system especially when dealing with large files through the file manager.

At 3:31PM UTC (06/01/2016) our Level 3 Data Center in Denver experienced a network outage that resulted in the full loss of service for nearly all services.

Timeline (times are UTC)

3:31PM Outage occurred

3:40PM Technician dispatched to verify local equipment

3:48PM Technician verified local equipment functionality

3:50PM Ticket opened with Level 3

4:05PM Ticket escalated to tier 1 support

4:22PM Ticket updated that Level 3 was investigating the cause of the outage

4:37PM Ticket updated to notify that equipment has been verified and that the fiber connections are good

4:42PM Ticked escalated to tier 2 support

5:20PM Ticket updated to notify that the connection is being reset/ rebuild

5:40PM Ticket updated to notify that our services should be back within 1 hour

6:38PM Ticket updated by ASPnix to notify that network was still offline

7:05PM Ticket escalated to tier 3 support

8:48PM Ticket updated to notify that the issue had been resolved and our services would be restored within minutes

8:51PM BGP connection reestablished on both ASPnix routers and service restored

Root Cause

ASPnix uses 2 private AS numbers provided by Level 3 to allow for BGP routing between our routers and the Level 3 edge routers that service our circuits. This allows for failover between routers in the unfortunate event that one fails. Due to an outstanding order that was tied to our account, their system automatically closed the order and marked our AS numbers as inactive. Within 5 – 7 minutes their routers received the updated configuration and notified that these routes are now inactive and were removed resulting in all traffic to and from our network to be unroutable.

Because of the missing AS numbers under our account, the tier 2 support technician assumed we did not use BGP and rebuilt our connection without it. Our routers are configured for BGP and advertise BGP, as a result they were unable to establish a connection and advertise our IP routes to the edge router.

Tier 3 support was able to see that our configuration was invalid and we provided our BGP information and previous AS numbers. The technician was able to restore the BGP routes and within minutes our service was back online.

Credits

We will be issuing credits to all customers in the next few days. Credits will be automatically applied to each account based on any active service(s). This excludes domains and service addons.

If you have any questions or concerns, please open a ticket to the technical support department for your service.

We’d like to thank everyone for their patience and understanding during this time and for choosing ASPnix!