Terrahost Statushttp://terrahoststatus.com/2019-05-25T15:05:01+00:00Terrahost Statushttp://terrahoststatus.com/incidents/69/We're investigating an issue with our Oslo fiber which seems to be down. We are in contact with our supplier to find the cause. No-one should be affected otherwise since traffic is being routed over our other transits.Update 23:00: Link to Oslo is back up. We are still investigating.Update 23:59: Confirmed the fiber optics is the issue and not our equipment in Oslo. Awaiting RFO from our supplier.
]]>2019-05-24T17:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/68/We have identified a failing switch in one of our racks causing issues for select customers. We are working to resolve the issue.
The issue have caused outage for some virtual servers, dedicated servers and colocation customers.Switch replaced and issue has been resolved.
]]>2019-05-20T11:30:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/66/We are experiencing issues with Debian 8 causing problems during install. The issue seems to be global and across all mirrors, where pxe boot files are missing. We are looking into the issue, in the mean time, Debian 8 has been disabled.

Update: Support for Jessie has been removed, as no more security patches are released, other than for LTS version. Jessie packages has been removed from official mirrors and moved to archive.

]]>2019-03-25T23:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/64/We are currently having issues with our mirror and LG-server, which are causing issues for server installations. Our techs are working to resolve the issue. In the mean time, OS installs will not work properly.

Update 22.02.2019 22:30: Issue has now been resolved.

]]>2019-02-22T15:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/65/A bug has been introduced into the netboot images of Ubuntu 16.04 causing it to halt during install.
Please see the following URLs for more information:
https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1817358
https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1816876

We are monitoring this and hopefully it will be resolved very soon. In the mean time Ubuntu 18.04 LTS works just fine.Update 28.03.2018: This has still not been resolved. Will close this for now. When the issue is resolved, we will enable 16.04 again.

]]>2019-02-22T03:59:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/63/Our fiber optic provider is planning maintenance on the fiber cable we use from our datacenter in Sandefjord to Oslo. The reason for the outage is a planned move of the fiber due to road construction in Lier. The outage is expected to last anywhere from 2 to 6 hours.

During this time our links to NIX, FIXO and our Blix transit will be affected.

Update 21.02: Planned work has been moved to another date.

]]>2019-02-20T23:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/62/We are currently moving backup accounts from SAN03 to a different storage. During this time SAN03 will be unavailable. We will update this case as soon as the accounts have been moved.Update: Move has been rescheduled to a later date. SAN03 is available again.
]]>2019-01-30T11:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/61/We are working to resolve the issue.Update 30.01.2019 15:00: Issue has been resolved. Console now works again.
]]>2019-01-28T22:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/60/We will in our upcoming maintenance window Thursday 24th of January between 23:00 and 06:00 move a few servers from old to new racks. This is part of our refurbishment of DC1. The new racks have dual monitored UPS power feeds and 10GE uplink to either one or multiple switches.

Customers with dedicated servers affected by this move are being notified directly. The move of servers is scheduled to take only 5 minutes, but since we need to power down all VMs and restart them after the move, expect some more downtime.

Update 23:30: The move is in progress.Update 01:52: All servers scheduled for the move has been moved to new racks. If you are experiencing any issues with your server, please let us know in a ticket and we'll see too it as soon as possible.

]]>2019-01-25T00:53:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/55/We are investigating an issue with our VPS Cluster.Update 12:50: We are working to restore a clean state on the storage cluster after the crash.Update 13:07: Servers are being brought up again. Expect to have all affected VMs up within 13:30.Update 13:19: Around 70% of servers has now been brought up again.Update 13:31: All servers has been booted.

Preliminary RFO
There are an watchdog issue on some nodes, causing them to fail fencing and reboot when an issue occurs in the network or cluster. These events are disturbing the whole cluster, since the storage is network distributed across all nodes. The Ceph storage cluster is quite resiliant and we have 3/2 replication. What this means is that all data on the storage is replicated at least 3 times, and we can loose up to 2/3 of the cluster, given the cluster has enough free space left to operate. But when multiple nodes reboot at once, a slowdown occurs. The cluster automatically booting 100 VMs at the same time does not help either!

We will look into how we can improve this and avoid reboots in the future. The cluster is now stable. Some customers may experience a slower performance for an hour or so while services are started .e.g on servers.

]]>2019-01-16T11:30:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/54/While working on our cluster to improve our services we experienced an unexpected reboot of VM Host pmx2dc2. We are investigating the cause.
]]>2019-01-11T14:45:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/53/We are investigating an issue with our VPS cluster causing an outage.Update 04.01.2019 19:12: We are continuing to investigate. Are able to reach all nodes in cluster. Storage seems to be okay.Update 04.01.2019 19:15: VMs are being powered up now.Update 04.01.2019 20:50: We are continuing to see instabilities. Investigating.Update 04.01.2019 20:57: A large DDoS attack is causing instabilities on our VPS Clusters management, causing hosts to loose quorum and reboot. We are trying to mitigate the attack and move our management to avoid it being affected.Update 04.01.2019 21:10: The attack has been mitigated. We are working to secure the infrastructure so that it can handle an attack without affecting other virtual servers on the same cluster. Servers are currently booting.
]]>2019-01-04T17:57:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/52/We have an issue with our API causing problems logging into our control panel and ordering services. We are looking into the issue and it will be resolved shortly.Update 31.12.2018 15:48GMT+2: Issue has now been resolved.
]]>2018-12-31T14:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/51/18.12.2018 08:57 GMT+1: We just witnessed a 10 second drop of our link to Oslo, causing BGP to converge as we lost transit and IXes in Oslo. We are monitoring the situation and are looking into it.Update 18.12.2018 10.07 GMT+1: Another drop just now around 10 seconds. We are investigating and checking with our supplier.
]]>2018-12-18T07:55:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/50/We are investigating a fault with a vm node.

Update 25.11.2018 09:24: node10.public-cloud seems to be offline. We are working to get the node up and running again

Update 25.11.2018 10:23: It seems to be a hardware failure. We are working on replacinh the faulty part.

Update 25.11.218 10:30: It has been confirmed that the faulty part (PSU) has been replaced and that the server is now working as it should. The server is shortly being put back into production.

Update 25.11.2018 10:39: Our techs are now booting affected VMs on the node.

]]>2018-11-25T08:32:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/49/We are currently investigating an issue where some installs does not start, or complete as they should.

Update: 23.11.2018 21.20 We see an issue where the DHCP does not respond to BOOTP requests in time. We are working to resolve the issue.

]]>2018-11-23T20:09:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/48/We are investigating an issue where some operating systems do not add customer SSH keys.Update 04.12.2018 09.41: This has now been confirmed resolved.
]]>2018-11-22T17:52:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/47/21.11.2018 09:30:
We are currently performing routine maintenance on one of our webservers, causing a few minutes downtime if/when the server is being rebooted.

21.11.2018 10:05:
Maintenance completed.

]]>2018-11-21T08:30:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/46/We are monitoring a virtualization host after an unexpected reboot today.
]]>2018-11-19T14:15:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/45/We where notified of power loss to one of our racks just now. A breaker had tripped and where immediately switched on again by our staff. This have caused some servers without dual power supplies to lose power for a few seconds and reboot.
]]>2018-11-08T11:45:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/44/We are currently performing routine maintenance on our Ceph Storage for the Proxmox cluster. Some may experience degraded performance.09.11.2018 We are finishing up maintenance for the Ceph Storage today. We have replaced a few slow drives in our cluster giving customers an performance increase.
]]>2018-11-07T09:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/43/We've currently experiencing another brief power outage. No virtual servers in either DC should be affected. Some dedicated servers in DC1 affected only.Update 13:10: Power has been stable but we are still running our generators in case of any more outages. We are investigating the cause.
]]>2018-10-02T11:45:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/42/Currently experiencing a second power outage today. We are investigating.Update 23:10: Utility power is back again. Dedicated servers and some virtual servers in DC1 has been affected by this. We are reaching out to our power company at the moment. Affected servers are being booted again. If you have a server in DC2 you are not affected by this power outage. Our datacenters are currently running on generator power.Update 23:30: We have confirmed that there is another issue with a 22kV line entering our area. One or two more power outages are expected within the next 1.5 hours to resolve the issue. Our datacenters is currently running of generator power and noone should be affected by the maintenance work by the utility company.Update 00:50: The power company has found the fault on the 22kV line and is currently working to fix the issue. It is expected that the power may go out again before it's resolved
Update 01:30: We are back on utility power and the issue should be resolved now.
]]>2018-09-27T22:50:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/41/We are currently experiencing a power failure to our whole block. Dedicated servers and some virtual servers in DC1 have been affected. No server in DC2 either dedicated or proxmox VPS should be affected. We are currently running on generator power.Update 11:06: If you have a VMWare based VPS in DC1, you may power on your own server via ENIGMA.Update 11:12: Utility power is back. We are currently keeping our generators running and we're monitoring the situation.Update 11:35: The following screenshot shows the power outage in the area: http://terrahoststatus.com/outage.pngUpdate 13:15: Our power company tells us that a 22kV line entering our county was damaged by a tree falling over the lines. The problem has been resolved and we're told there will be no more outages.
]]>2018-09-27T10:50:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/38/We're experiencing an issue with one of our VPS nodes. We are investigating the issue.Update: We are monitoring the situation. It is stable for now.Update: Issue has been resolved.
]]>2018-09-19T18:39:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/37/We are currently experiencing an issue with one of our Proxmox nodes (pmx9). We are looking into the issue. If you server rebooted just now it most likely was on this host. All VMs was automatically migrated to another host and restarted.
]]>2018-09-18T13:10:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/36/We are currently doing some required maintenance on the storage cluster used by Proxmox VPSes. This are causing some degraded disk performance. It's estimated that the maintenance will take around 30-45 minutes and finish around 10:30 GMT+2. We are sorry for the inconvenience. Further work on the cluster will be scheduled to night time GMT+2.Update 10:25 GMT+2: Maintenance work is now completed.
]]>2018-09-13T09:30:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/35/We are currently having an issue with our Debian Mirror causing issues for software installations and new OS installs. We are working on resolving the issue and expect to have the mirror up very soon.Update 09:15: Issue has been resolved.
]]>2018-09-06T09:02:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/34/We are experiencing issues with our fiber optic link to the HMG9 datacenter in Oslo causing BGP to flap. Interface has been shut down and our suppliers techs are working to find and resolve the issue. No customer should be affected by this, as traffic is being routed through our other two transits.Update 16:57: Our signals received in Sandefjord is a bit low and may cause this issue. Our techs are continuing to investigate this case. At the moment the link is up.Update 3/9 09:13: This issue has now been resolved.
]]>2018-08-31T10:35:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/33/We are currently performing updates to our KVM cluster for virtual servers. The cluster is configured HA and no downtime is expected while we upgrade our nodes. Affected nodes will have their VMs migrated away before upgrading. This case will be updated once the updates are completed.Update 23:00: Updates has been completed. No VMs should've been affected.
]]>2018-08-30T13:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/31/We are currently experiencing periodic outage on our website and API functions. We are looking into it and will update this incident once we know more.
]]>2018-08-30T08:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/32/We are currently experiencing an issue with our mirrors. This is causing problems for OS installs. We are working to resolve this as soon as possible.Update 31/8 11:29: Issues with mirrors causing troubles for OS installs and package installs has been resolved.
]]>2018-08-29T00:30:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/30/Maintenance is currently being performed on our fiber to Oslo. The link link is estimated to be down between 0100 AM and 0600 AM GMT+2 (August 15th). Traffic is being routed between other transits in the mean time.Update 15/8 12:00: Maintenance has completed and the link is now up again.
]]>2018-08-15T01:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/29/We have discovered that in some cases virtual servers may have troubles rebooting properly to initiate PXE boot and our reinstall procedure. We are looking into it and will update this case once we know more. If you are experiencing issues with your server not starting the install, you can go back in ENIGMA and reboot it manually.Update: Issue has been resolved. It was due to some VMs having an error state in HA.
]]>2018-08-13T20:55:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/28/We are experiencing issues reinstalling a few operating systems. Our techs are looking into it. This case will be updated once we know more.Update 02/08 19:30: Issue seems to be isolated to Ubuntu 18.04. We are investigating, most likely a corrupt mirror.Update 03/08 01:00: Issue has been resolved.
]]>2018-08-01T08:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/27/We have just discovered an issue with a SAN in our network effectively causing mirror.terrahost.no to go down. This will cause application installs/upgrades to fail and new installs to halt. We are investigating the issue.Update 15:15: Issue has been identified and we're working on a fix. Should be up shortly (10-20 min)Update 15:40: Mirror is back.
]]>2018-07-26T14:55:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/26/We are experiencing an issue with one of our distribution routers. The issue affects parts of DC2. We are working to resolve it as soon as possible.Update 00:30: Techs are on site. Expecting to have the distribution router up within a few minutes.Update 00:43: Router is expected to be back up again before 01:00 GMT+2Update 01:06: Router is available again. We are running checks to make sure this does not happen again.Update 01:18: Quick RFO: The reason for the partial outage in DC2 was caused by a configuration error causing the full routing table to be injected into OSPF. This caused numerous errors due to the fact that some routers are unable to handle this table size in IGP. After clearing the OSPF processes and rebooting the router to clear memory it came up as it should. We are sorry for the inconvenience.
]]>2018-07-20T00:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/25/We are currently experiencing some issues with our Blix transit. The issue should be resolved shortly.Update: The issue was resolved as of 8AM this morning (GMT+2).
]]>2018-07-18T20:32:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/24/We will be performing an firmware upgrade of our distribution router sfjdist1-dc2. There is an estimated downtime of 30 to 120 seconds.
]]>2018-07-09T01:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/23/We are currently performing necessary updates to our internal webservers. ENIGMA and our websites may briefly be unavailable.
]]>2018-07-02T13:30:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/22/We've received reports of issues with enabling 2FA in Enigma with the Android version of Google Authenticator. We are looking into this and hope to have it resolved very soon. IOS users are not affected.
]]>2018-06-25T04:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/21/We are currently experiencing an outage on our link to Oslo. The operator is investigating the issue. Estimated time of repair is 18.00 CEST.Update Sat 20:00 GMT+2: ETA has been moved to 20:00 CESTUpdate Sun 01:30 GMT+2: The damaged fiber optic cable has been repaired but seem unstable. Link is going up and down causing BGP flapping and issues. We are currently administratively shutting down the interface until we know it's okay to open again. Our provider estimates the cable to be completely repaired within 03:00 GMT+2.Update Sun 02:30 GMT+2: The link is stable and brought up again.Update Sun 14:45 GMT+2: We are again experiencing errors on the link causing BGP flapping. Although it should not affect our normal operations, we are shutting down the interface util everythings cleared.Update Mon 12:00 GMT+2: Our provider confirms that they are having issues which they are working on to resolve. As for now the port remains shut down.Update Tue 12:00 GMT+2: The link is now up again. Our provider will perform a replacement in August to fix the issues we've been having permanently.
]]>2018-06-23T09:30:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/20/There is currently an issue with paying invoices with PayPal either on auto-billing or manually. Customers cannot register their PayPal account either. The problem has been located and we're working on a fix.Update 14:13: We have confirmed that our merchant (Braintree) has issues at their end. They are working to resolve it as soon as possible. It may take 1 - 3 days before payments with PayPal is available again.Update 18/5 08:41: Our merchant is still experiencing issues that they are working to resolve. PayPal payments should be available again later today.Update 18/5 17:00: PayPal payments is now back again.
]]>2018-05-16T08:28:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/19/This morning 3rd of May around 08:00 GMT+2 we are moving the following subnets to GLBP:
185.125.168.0/24
185.125.169.0/24
185.125.171.0/24

The move should not be noticable as the subnets are expected to be up again after 2-3 seconds.

Update 08.25: All subnets has now been moved to GLBP. Outage was under 3 seconds per subnet.

]]>2018-05-03T08:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/18/We are tonight (0100 - 0200 GMT+2) moving a few access switches to a new distribution router. This may cause 10-40 seconds dropout. If you are experiencing a drop in connection to your server, it should be back in just a few seconds.Update 01.32: All access switches has been moved. An average of 5 seconds dropout was experienced.
]]>2018-05-01T01:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/17/We have identified an issue with one VPS host. We are working on bringing affected VMs back online. ETA 10-15 minutes.Update 18:00: Server is currently booting, ETA 5 minutes.Update 18:05: We are working on powering on VMs.Update 18:13: All VMs are in the process of powering up and should be available shortly.Update 20:15: We have rebooted a few VMs that did not boot properly.
We have identified the issue causing this host to fail. Parts have been ordered and we will update this case when we will upgrade this host the coming week.

Update 3/5-18 20:30: This VPS host is down again. We are waiting for parts to permanently fix the issue. Techs are working to get the VPS host back up.Update 3/5-18 20:35: We are working on powering on affected VMs. All servers should be back up shortly.Update 3/5-18 20:58: All VMs have been powered on. If your server is not responding, please try to reboot from our control panel. The root cause to this VPS hosts issues are the CPUs. We are waiting for new CPUs which should arrive either tomorrow or just over the weekend. In addition to this, we will also BIOS upgrade the server.Update 7/5-18 10:03: Necessary maintenance on this VPS host will be performed tomorrow at 4 PM GMT+2. More information in the planned maintenance.

Update 8/5-18 15:40: We have begun graceful shutdown of VMs on this host in preparations of the CPU replacement and BIOS upgrade.Update 8/5-18 15:57: CPU and BIOS upgrade is completed. We're currently booting the server and will boot the VMs in order.Update 8/5-18 18:00: All VMs have booted. Some VMs have needed an extra reboot. This node should now be stable.

]]>2018-04-29T17:45:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/16/We have today identified an issue with our bandwidth monitoring systems, causing a large amounts of emails regarding bandwidth overusage to be sent out. We are looking into the issue.
]]>2018-04-01T01:54:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/15/We've identified an issue with one of our VM nodes. Techs are working to resolve the issue as soon as possible.Update 07:45: VM node is up and all VMs should be in the process of booting. We are checking logs to find the root cause and to make sure this does not happen again.
]]>2018-03-12T05:50:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/14/We are currently experiencing issues with our Altibox transit due to a damaged fiber optic cable. Our techs are currently repairing the damaged fiber. Customers should not be affected by this other than traffic entering / leaving our network on transits other than Altibox. It is estimated that the repair takes 3 - 4 hours.Update 15:15: Our techs are still working on repairing the damaged fiber optic cable. We are estimating that it's up within 1-2 hours.Update 16:17: Fiber cable has been repaired and Altibox has been reactivated.
]]>2018-03-07T12:30:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/13/We have identified an issue with our system monitoring bandwidth overages and email notifications. Few customers has today received notices about incorrect bandwidth overages by mail, and some quite few of them.

The issue has been located and resolved. You should not receive any new emails regarding this.

]]>2018-03-01T08:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/12/We are currently experiencing some sporadic packet loss on our Altibox transit. Our techs are investigating the issue.Update: Issue has been resolved.
]]>2018-02-19T19:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/11/Feilen er funnet og det jobbes nå med å rette feilen.
The error has been found and it is now working to correct the error.

We are working on some of our VPS nodes that have lost networks. The case will be updated as soon as the error has been found.

Update 14:50: We have identified the issue and virtual servers on affected nodes should be back online. We're sorry for the inconvenience.

]]>2018-02-19T12:53:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/10/We're investigating an issue with storage on one of our VPS nodes (node4-pc2).Update 01:00: We are migrating servers off the affected storage to investigate further. VMs have been shut down.Update 01:20: We're continuing to migrate VMs of affected storage.Update 01:23: The node will be rebooted within 20-40 minutes to allow for faster migration of the remaining VMs on the affected storage. All VMs on this node will be shut down briefly to perform the reboot.Update 01:51: Graceful shutdown of remaining VMs is being executed. Node set to reboot within 1-2 minutes.Update 02:29: Migration complete. Will we will check the storage tomorrow morning and replace it.
]]>2018-01-30T23:50:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/9/We have received reports of IPv6 not working for some customers, unable to reach gateway. We are looking into it and hope to have it resolved very soon!

Update 22:22 GMT+1: As we're continue to investigate what the issue is with our DHCPv6-PD deployment, we have issued all servers with our new IPv6 statically routed IPv6 platform. This platform was meant to be launched later this quarter, but has been moved up to allow customers having issues with IPv6 to quickly migrate and get back up again.
We ask that customers having issues with IPv6 please log into their control panel and find your new subnet. It will be marked as static rather than DHCPv6-PD.

Update 29.01.2018 09:54: Most customers has migrated away from the DHCPv6-PD platform. We will soon close it down. If you are having issues with IPv6, please contact us. Your new subnet has been assigned.

]]>2018-01-25T18:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/8/We're experiencing a few issues with our core routers causing BGP sessions to flap. Issue has been identified and we are working to implement a fix.

Update 10:20: We are currently power cycling one of our core routers. ETA 10-15 minutes.Update 10.34: Core router has been power cycled. ETA 5 minutes.Update 10:59: Issues with NVRAM on one of our core routers. Manually restoring config backup.Update 11:25: Backup has been restored. We are proceeding with supervisor replacementUpdate 11:40: We are verifying switch uplinks. Had issues with a few switches. Should be resolved now.Update 11:56: We are seeing some packet loss in parts of our DC. Working to resolve it.Update 12:01: IOS has a self defense mechanism when low on RAM it inserts a limit into the TCAM causing routes to be dropped. This limit was the reason for the sporadic packet loss at certain parts of the DC.

Update 13:30: We are seeing similar issues now. The team is working on replacing the router.Update 13:37: CPU and MEM is normalizing.Update 14:50: Old supervisor seems to have stabilized. We have moved one transit to the new Supervisor.Update 14:53: Now preparing to move Telenor transit to the new Supervisor.Update 15:02: Telenor BGP session has been GSHUTUpdate 15:11: Telenor BGP session has been brought upUpdate 15:22: Finalized shutdown of old supervisor

The issues with our core router should now be resolved. We are sorry for the inconvenience.

]]>2018-01-24T09:33:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/6/We've identified an issue with one of our core routers which caused BGP sessions to flap. Investigation is ongoing and we are monitoring the situation.Update 11:25 GMT+1: We are still monitoring this. Measures are put in place in order to avoid this issue in the future.
]]>2018-01-18T10:15:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/5/When a peer dropped we experienced an issue with long BGP convergence time and issues with some routes to/from our datacenter. Some may have experienced this as complete loss of connection to our DC, and others not. The issue has been resolved.
]]>2017-12-13T12:50:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/4/We're experiencing an issue with one of our VPS nodes. We are investigating the issue.Update: We are powering off the node for a moment to investigate furtherUpdate: Issue is related to storage. We're booting the node and moving affected VMs to another storage.Update: Unaffected VMs is currently booting.Update: Affected VMs are now being moved into new storage. May take some time.Update: The failed storage is slow and it will still take some time before all VMs are back to normal.Update: About 50% of affected VMs are back to normal.Update: All VMs are now back online.
]]>2017-12-08T13:15:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/3/We have had a few issues with one of our storage SANs causing problems for a few services. Some web servers where affected, our spamfilter and ASP Exchange. Affected services are on their way up again and should be back shortly.
]]>2017-11-25T18:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/2/We've identified an error in our Debian 9 installation procedure causing some servers to remain offline after install. Issue has been resolved.
]]>2017-11-24T14:00:00+00:00Terrahost Statushttp://terrahoststatus.com/incidents/1/We are experiencing issues with a VM Host and one of it's storage arrays. We are working on recovering the host and will be migrating affected VMs later today to new storage.
]]>2017-10-26T15:00:00+00:00