POWERED DOWN: Flood takes out AAPT datacentre

The Queensland floods have claimed their first datacentre victim, with AAPT’s facility at 167 Eagle St in Brisbane confirmed to have powered down. And a number of other facilities may be about to follow it, as the statewide disaster continues to hit technology providers where it hurts.

AAPT this morning distributed a notice to customers stating the 167 Eagle St facility had gone offline. “Due to the flooding of the Brisbane region, AAPT will power down the equipment at the 167 Eagle St, Brisbane site as a safety requirement,” the telco said in a statement. Sources added the telco’s other facilities at 344 Queen St and up the road at 123 Eagle St were also expected to go down shortly.

A spokesperson for the telco this morning was unable to be contacted on the matter, and technicians at AAPT’s network operations centre declined to comment, saying they weren’t authorised to do so. In its statement, AAPT said it was currently investigating the issue and would provide further information once it became available. It apologised for the inconvenience caused.

It is unclear how large and how critical the AAPT facilities are to the running of the state’s telecommunications systems, although the 167 Eagle St facility is believed to be a major one, in the heart of Brisbane’s financial and legal district. And AAPT may not be the only company to suffer problems with its datacentres in the wake of the floods hitting the state capital. PIPE Networks may also shortly be facing power shortages at some of its Brisbane facilities.

The company has sent out a string of notes to clients this morning. At first it noted that it did not expect any problems, but later operations manager Matt Whitlock noted in an email that emergency services agencies and energy utility Energex had advised the company that it may be necessary to disconnect its 127 Creek St facility from grid power.

“Currently the concern is regarding the substation located in the lower basement of the building,” wrote Whitlock. He noted that if power was cut to the facility, PIPE’s UPS systems and generators would carry on delivering power to its DC1 and DC2 facilities, with its DC3 and DC4 facilities continuing to run on mains power.

“We have confirmed that there is sufficient fuel to operate the building for at least 24 hours,” he added.

However, later in the morning Whitlock noted the situation had changed, and PIPE had been advised that its DC1 and DC2 facilities at 127 Creek St would be “one of the last buildings to be disconnected from the grid by Energex” as it housed a critical Qld Government department coordinating the State Emergency Service response to the flood.

“Energex have advised that if the flooding reaches 15-20cm of standing water in the lower basement car park of 127 Creek St, all power (including the generators) will be cut to the building,” wrote Whitlock. “This will result in DC1 & 2 going dark until such time as Energex advise us that power can be safely restored to the building.” DC3 and DC4 won’t be affected, however.

And this could be a problem for PIPE, given the company’s previously mentioned 24 hour time frame.

“Worst-case scenarios puts this time at 48 hours,” said Whitlock of the potential outage from the grid power system. PIPE has now evacuated the 127 Creek St office — including its network operations centre, although it continues to monitor its network remotely. “As I’m sure you can understand, we are working in a very changeable climate and trying to react to the changes in advice,” wrote Whitlock.

Not everyone’s coming so close to the wire. Melbourne IT spokesperson Tony Smith said his company’s datacentre was in Wickham St in Fortitude Valley, on the edge of the central business district. It currently had full power from mains supply, and obviously generators for backup, he said.

“We’re going to keep monitoring the situation, and trying to get in touch with Energex, to see what their plans are. We’ll essentially be ready,” Smith said.

Maybe people should get their facts straight before publishing their articles for everyone to see – there isn’t an AAPT datacentre in that building (167 Eagle Street), only equipment that services customers who are located in that building who may lose service as a result of flooding.

Just shows how ignorant some people are and how they would like to market their businesses…

Based on this, I would strongly suggest that mains power will be shut down.
Even though this facility has UPS and backup diesel generators, they will no doubt put the facility into shutdown due to significant threats not to mention the OH&S risks.

The Eagle St location seems to only be a in building POP servicing building tenants and would therefore have no significant impact on the operation of the AAPT network either nationally or within QLD.

PIPE Networks has two locations in QLD and both of these locations are listed as threatened by flooding or within a very short distance to flooding
– 127 Creek St, Brisbane (PIPE DC 1 & 2)
– 148 Brunswick St, Fortitude Valley (PIPE DC 3 & 4)

If PIPE was to shut down both QLD locations, then this would have a very significant impact on PIPE’s network in QLD

This site is not actually listed by AAPT anywhere as a customer facility, woudl suggest a network POP. Battery power will not hold the facility indefinitely.

UPDATE:

——–
AAPT Alert NOTIFICATION: SERVICE PROBLEM:

Window : 13:57 AEDT Wednesday 12-Jan-2011
Event : Outage

Affected Services : All Services in QLD in all POPs

Message:
AAPT Reference – 1040404

Initial Notification

Location Affected – 123 Eagle St, Brisbane (Riverside Colo)

Network at fault – Transmission

Services Affected = IP, VOICE & DATA, INTERNET, ETHERNET

Additional Information – Due to the flooding of the Brisbane region, mains power to the facility at the Riverside Centre site has been shut down as a safety requirement. Site is currently running on battery backup.

There are three critical requirements for ANY Data Centre, this is well known, even in EU and US as a must…

They must never be located in:

1 – An area prone to earthquakes
2 – An area prone to flooding
3 – An area near any critical infrastructure that may be deemed a target by terrorists

AAPT and PIPE both *FAIL* in #2 and although close by, I deem only AAPT to also fail in #3, those who know that area know why I say this.

Subsequently, ISP’s colocating their equipment with PIPE, are as much to blame in #2.

Back in ’06 127 creek st had a full power outage due to a fire condition (which mandates NO generator operations), most ISP’s customers in Qld went byebyes as all relied upon PIPE, because everyone placed their eggs in one basket, only AAPT/Powertel and Veridas customers remained online IIRC because neither soley relied upon that facility for core operations.

I think your header is slightly misleading with use of the term “Datacentre”. The site in question is used by AAPT as a POP site, you’ll find a lot of gear used for transmission but you wont find customer gear in there.

Also it’s significant but I dont think this is as big a deal for them as you article is pointing out, they have full redundancy via the Valley.

The company has redundancy however FaktorTel VOIP provider has a interconnect running directly into that POP. With out a second interconnect into their second site the DID’s are down.

Our VOIP and they route the national numbers (Brisbane, Sydney, Melbourne) through that location.
They will be working with AAPT to move the interconnect to their redundant site but its still down and has been for hours.

Comments are closed.

Book now available

Written by Delimiter Publisher Renai LeMay, The Frustrated State is the first in-depth book examining of how Australia’s political sector is systematically mismanaging technological change and crushing hopes that our nation will ever take its rightful place globally as a digital powerhouse and home of innovation.

Welcome! We were an energetic and engaged community of Australians who worked with or who were interested in technology -- all sorts of IT professionals, IT managers, CIOs, tech policy-makers and tech enthusiasts.