Telecom advises that it has restored service to the vast majority of customers, but is monitoring the network closely and urgently investigating the underlying cause of the problem.

Telecom's CEO Retail Chris Quin says customers expect a reliable and robust broadband service from Telecom and rightly so given the role the internet now plays as a major communications tool.

"We have a comprehensive review underway to ascertain how these issues can be minimised in the future and apologize unreservedly to customers for any disruption these outages caused in the interim."

Telecom apologises to customers affected by these incidents and the inconvenience they have caused.

The first outage occurred early yesterday morning with service restored by 8.15am.

Another outage last night was identified at 9.45pm with service restored by 11.35pm (although it took up to an hour or more for some customer modems to be reconnected).

The first outage related to a problem with a software upgrade on an Auckland-based server that is part of a system which handles just over 300,000 customers, representing approximately two-thirds of Telecom?s broadband customer base. The problem meant customer modems were failing to authenticate when trying to communicate with the server. The Auckland server normally runs in parallel with a Christchurch based server and service was initially restored by removing the Auckland server from the system and running solely on the Christchurch server.

Yesterday afternoon service resumed on both parallel servers, but last evening further problems were experienced with the Christchurch server overloading due to issues synchronising information between the two servers. To address this issue, the Christchurch server was taken off the system late last night, with internet traffic all going through the Auckland server.

Close monitoring overnight indicated that some customer modems were having intermittent problems connecting to broadband, which is believed to have been caused by the capacity pressure on the single Auckland server. The Christchurch server was brought back up just after 9.00am this morning and the number of internet connections improved rapidly from this time, with 90%+ active connections by 9.30am.

Telecom helpdesks are also experiencing higher than normal levels of calls for this time of the day, but this may be due to customers being unable to connect because they need to reboot their modems first. Although most modems will reconnect automatically, a minority need to be turned off and back on again.

I don't know if its related but in the Auckland CBD last night around 1:30am -> 2:30am Telecom Mobile looked like it was out. No voice calls could be made in/out but data was working until about 2:00am and then stopped the same as voice until 2:30am.

LennonNZ: I don't know if its related but in the Auckland CBD last night around 1:30am -> 2:30am Telecom Mobile looked like it was out. No voice calls could be made in/out but data was working until about 2:00am and then stopped the same as voice until 2:30am.

Richard from Telecom here, appreciate the thoughts on the service alerts. For us, we're trying to explain why your broadband is down, in language that everyone can understand. Also understand some of the more tech savvy folk here may find them lacking in detail, we take that on board.

Lastly, apologies to everyone here who was affected. This last 24 hours or so is the last thing we're trying to provide you - yes, the team here are working extremely hard to get this sorted out, as you'd expect.

richirvine: Richard from Telecom here, appreciate the thoughts on the service alerts. For us, we're trying to explain why your broadband is down, in language that everyone can understand. Also understand some of the more tech savvy folk here may find them lacking in detail, we take that on board.

Lastly, apologies to everyone here who was affected. This last 24 hours or so is the last thing we're trying to provide you - yes, the team here are working extremely hard to get this sorted out, as you'd expect.

Cheers

Thanks Richard for the update. You're on Geekzone so I think it's safe to use technical speak if you want to. :)

richirvine: Richard from Telecom here, appreciate the thoughts on the service alerts. For us, we're trying to explain why your broadband is down, in language that everyone can understand. Also understand some of the more tech savvy folk here may find them lacking in detail, we take that on board.

Lastly, apologies to everyone here who was affected. This last 24 hours or so is the last thing we're trying to provide you - yes, the team here are working extremely hard to get this sorted out, as you'd expect.

Cheers

Thanks Richard. For people reading here, the updates in the media page are the same ones I am posting here as soon as I receive them.