Updates, issues and planned works

Unplanned downtime

We are aware of a problem with authentication to our Radius servers in the last 20 minutes that is affecting ONLY circuits delivered as from C20 exchanges — we are working on this now and expect a resolution shortly.

It’s only affecting those lines that have dropped earlier or had been switched off and are now trying to connect (“authenticate”) and not on lines currently logged in all of which should carry on working normally; so is only a problem for a small part of the ADSL estate in Merula.

[Update] 15:45 The engineers have repaired the faults identified and all circuits are up & reporting as being heathy. If anyone is still seeing issues please mail into support@merula.net. We are conducting a postmortem with the supplier to ascertain the root cause, the fix made and to find out why their agreed backup routing didn’t kick in as part of the DR process. We will update the ticket as we know more.

15:10 Multiple back-hauls to us and other customers are affected by this fault; this has been escalated internally by the supplier to their 3rd-level team. No time to fix as yet.

14:26 Engineers have arrived on-site and are starting their investigations.

Engineers are en-route to both ends of this connection.

We are aware of a problem affecting one of our core bearer lines into HEX in London which in turn is affecting a number of customer leased lines. These are currently hard down.

This is a high-priority outage for us and we are working to get this resolved as quickly as possible and apologise for the down-time being seen. We do not currently have a time to fix but expect a status report within the next 30-45 minutes.

One of our Virtual servers currently has a failed RAID. The drive is rebuilding with a replacement drive.

Sites currently hosted on this server are currently unavailable – this includes one of our primary name servers (which should have no affect) and a couple of customer servers along with one of our shared web sites.

We are working to bring this back as soon as possible and will update this as the disk re-build progresses. We believe at this time that no data has been lost

This does not impact any services UNLESS your site was hosted on the affected server.

For a brief period, the RADIUS server here in Huntingdon that authenticates ADSL logins was refusing connections. This is now resolved and any ADSL circuits that weren’t able to login should have automatically come back on line again. If you are still seeing issues, please re-start your router.

We had to reboot one of our core ADSL routers as we were aware of various problems affecting a significant number of lines and we took the decision that a very short outage was needed to allow these sessions to be cleared down. We have seen all of the affected lines come back successfully, so believe that this is now resloved. Apologies to anyone who briefly lost their connection.

UPDATE: the fault has been isolated and fixed. Any customer facing services affected should now be back to normal.

We’ve a hardware issue that is affecting some hosted WP sites and may impact other sites also hosted on the same server. No time-to-fix information as yet; we’ll update this post as more information becomes available.

We are aware of a network issue that occurred this morning. This appears to have been caused by one of our routers crashing.

We do have multiple routers so normally this would not cause a problem – however in this case it would appear that while the router stopped routing some traffic properly it was still advertising the route to our other routers.

We have now reset the failed router which so far seems stable – we are monitoring the router for any signs of issues