How the sewage plant broke

Halifax’s treatment plant failed seven months ago Friday, but only now do we know what happened. For the first time, The Coast examines exactly what went wrong, why it went wrong, how to fix it and the lessons learned. Now, with video!

[Editor’s note: this story is one of five Coast articles selected as finalists for the 2010 Atlantic Journalism Awards. All five stories are collected here.]

Halifax's new sewage treatment plant was turned on in
February 2008, and it seemed to fulfill its promised intentions
immediately. All you had to know was that parts of the harbour were
swimmable again, and the most expensive infrastructure project in
Halifax history seemed worth it. By last January, the plant had been
operating almost a year, facing every weather condition without a
hitch. There was no reason to think the heavy rain forecast for
Wednesday, January 14 would prove to be the plant's undoing.

Until recently, we had no clear idea what exactly went wrong---mayor
Peter Kelly has hidden information on the plant failure behind a wall
of secrecy. He has refused to release a $100,000 forensic audit
investigating the failure, and Tuesday he scheduled yet another
closed-door, no-paperwork council meeting to discuss the sewage
situation.

But over the past several weeks I've interviewed engineers from
cities across North America who have explained how their sewage systems
work, as well as many local sources who are familiar with the design
and construction of Halifax's system. Additionally, officials with the
Halifax Water Commission, who are responsible for fixing the plant,
have recently become more talkative, and are now answering direct
questions. Together, those conversations have led to a deeper
understanding of the plant failure.

What went wrong

Heavy rain alone wasn't enough to break the plant. In the early
hours of that January morning, a series of cascading mechanical and
electrical problems occurred---a perfect storm of errors. The disaster
started with a Nova Scotia Power failure throughout the north end of
Halifax, including the plant.

The treatment plant is built around an 85-foot-deep "wet well." A
large tunnel carries Halifax's sewage---toilet flushes, rainwater,
anything running through the city's ancient sewer pipes---to the bottom
of the well. Four submersible pumps (and a fifth backup) at the bottom
of the well lift the sewage up to the treatment equipment on the main
floor. When the plant lost its power and the pumps stopped working, a
large iron gate automatically closed over the tunnel, stopping the flow
of sewage into the wet well.

Twenty minutes later, an on-call technician arrived to fire up two
backup generators. With the generators online, the technician opened
the gate and the four pumps began operating. So far, the plant had
responded to the power outage as it was designed to.

But the electrical load from the pumps was not evenly shared by the
two backup generators. One generator carried the load for three pumps,
while the second carried the load for only one pump. The generator
carrying the three pumps overloaded and shut itself down, leaving just
one pump to handle all be rainwater and other sewage coming into the
plant.

The plant couldn't last long with just one of its four pumps working
off a backup generator.

To stop sewage from coming into the wet well, the gate should have
closed over the tunnel, but the mechanism for lowering the gate was
also powered by the overloaded generator and so didn't work.
Anticipating just such an emergency, the plant has a switch designed to
shift the gate's load to the second generator, but this morning it
failed to operate properly. The gate was therefore left slightly open,
and sewage continued to flow into the wet well.

The pump mechanisms and motors are in watertight casings, so they
weren't in any danger from the rising sewage. However, they are powered
by cables leading from electrical junction boxes placed just 10 feet
above the pumps. The boxes are not watertight. When the rising sewage
reached the junction boxes, it flowed into them, down the electrical
conduits and into the pump casings, shorting out each of the pump
motors, including the motor running the lone working pump. At this
point, even if the power came back on, all the pumps would be
useless.

With no working pumps and the tunnel gate open, sewage continued to
flow into the wet well, rising all the way to "hydraulic grade"---the
level where water pressure evens itself out, in this case sea level,
which is roughly at the ceiling of the plant's basement. Much of the
plant's equipment, including the electrical control room and boilers,
were in the basement, below hydraulic grade, and therefore completely
immersed in sewage.

Raw sewage has been flowing into the harbour ever since. City
officials say they can put the plant back in working order by spring of
2010.

How to fix it

Placing the pumps' junction boxes and the electrical control room
below hydraulic grade was a tremendous design error.

Carl Yates, manager of the Water Commission, tells me that he'll
have to completely rebuild the electrical control room. There isn't
space to move the new equipment up to the main floor of the plant, but
he's confident he can isolate the room from the wet well---that is,
he'll seal off the room, have its drainage system lead away from the
wet well and lift the electrical cables up over the walls, above
hydraulic grade, before descending again into the wet well.

"Lots of operations are set up like this---look at all the plants
around harbours," he says. Pressed for a specific example, he points to
Toronto, which like Halifax has deep pipes feeding its sewage
plants.

Indeed, many cities, including Toronto, Boston and Portland, Oregon,
have similar "big pipe" sewage systems, but none of those cities favour
using submersible pumps or placing electrical equipment beneath
hydraulic grade. Instead, each uses a "wet well/dry well" design---the
sewage flows into the wet well, but the pumps and electrical equipment
are kept safely isolated in an adjacent dry well.

Toronto's system does have some electrical equipment below hydraulic
grade, explains Frank Burford, senior engineer with the city of
Toronto, but those are in plants that were designed at least 50 years
ago.

"In new designs all the electric is at grade or above, so you keep
your electrical stuff out of that environment," he says. Moreover, with
those older inherited designs, Toronto is moving to rebuild the plants
to move equipment above hydraulic grade.

Would Burford design a new system with electrical equipment below
hydraulic grade?

"No," he says unequivocally.

Yates, for his part, says the junction boxes in the Halifax plant
will be moved above hydraulic grade.

"Those will absolutely come up," he says.

I had that conversation with Yates on Friday afternoon. Tuesday,
after the closed-door council meeting, the city issued a press release
stating that "all electrical junction boxes will be moving up to the
street level area of the plant with the delivery of longer cables
required to do this expected by the end of August."

In addition to the design error of placing electrical equipment
below hydraulic grade, there were also at least three problems
associated with the backup generators in the plant: the generators'
powering to the pumps was improperly sequenced, the gate mechanism
switch between the generators failed and the generator overloaded. It's
unclear at this point if these were strictly design errors or if the
equipment was incorrectly assembled, or both.

One question that needs to be addressed: Why was an emergency
generator that shuts itself down when overloaded put in the plant? Many
emergency generators---for example, those that power fire fighting
equipment---are designed specifically not to turn themselves off
when they overheat; it's judged better to risk losing a generator than
to lose power to the equipment the generator is attached to.

Regardless, it should be a simple matter to right these
generator-related problems.

Lastly, there's the tunnel gate, which evidently could not be closed
manually, another clear design error. A weighted, manually operated
gear assembly will need to be installed, so that a technician can close
off the tunnel, even when there's no power in the plant.

Again, after discussing the gate issue with Yates last Friday, the
city's press release on Tuesday reflected the gist of our
conversation.

It reads, in engineering lingo: "The sluice gate actuator has
arrived and is being assembled with the gear box."

Lessons not Learned

It's tempting to treat the sewage plant failure as merely a narrow
technical problem---discover the mechanical issues involved, fix them
and be done with it.

That's certainly the view of mayor Peter Kelly, who has repeatedly
said we should "look forward" and "move on" to next summer, when the
plant will again be operating correctly and the harbour once again
clean enough to swim in.

That approach has the added benefit of taking the public focus off
assigning fault for the failure, a politically messy matter of
contention that will no doubt end up in a multi-million-dollar court
battle.

But the sewage plant failure is not just a narrow technical problem.
Rather, the technical problems at the plant were the result of
political and bureaucratic decisions made at city hall years
ago---process issues.

Ignoring those process issues, or hiding them behind a wall of
secrecy, will mean that they won't be properly addressed, and we'll
have other catastrophic or costly events in the future.

At its root, the sewage plant failure is a failure in how government
gets stuff built.

Traditionally, when city governments undertook large projects,
they'd hire an engineer to design the project, and then contract with a
construction company to actually build it. The engineer, employed by
the city, represented the city and closely watched to make sure the
contractor built the best possible project. After construction, the
city would take ownership of the project, and city employees would
operate it.

But in recent decades cities have increasingly contracted large
projects out entirely to private businesses---that is, the design and
construction (called "design/build") and sometimes additionally the
actual ownership and operation (called "design/build/operate") of the
project is undertaken by private companies.

Halifax's sewage system was built through a design/build contract
with a consortium of two private firms---Dexter Construction and
Degremont Limited.

"To properly do a design/build---to actually make it work---you also
have to give the operation component of the system to the private
firm," says Frank Burford, senior engineer for the city of Toronto, who
worked most of his career in the private sector. "Because otherwise the
private companies will do a design/build that is most economical for
them---because that's what it's all about, in money---and so redundancy
and extra features that will make it more reliable to operate long-term
can be sacrificed, because their costs end when they turn the keys over
to the owner.

"I found that design/build in Ontario didn't really work," he
continues. "You really do need the fingerprint of the owner on the
thing, because then they can review it. It takes maybe a little bit
longer to get something done, and it might cost more, but there's more
checks and balances."

Burford points out that whatever cost and time savings Halifax
gained by going the design/build route for the sewage plant were lost
with dealing with the plant failure.

He's ambivalent on whether cities should stick with the traditional
method of hiring an engineer to represent them in a building contract
or go the opposite direction and contract out in design/build/operate
fashion. Either way, he insists, "operations guys" must be involved in
the design process.

"You get together with the designer of the facility, your operations
people and your engineers in the city," says Burford, "and you look at
'what if'---what if this goes wrong, what if this fails, what kind of
contingencies do we have? Then you look at all the Murphy's Law
stuff---what if, what if, what if, and are we covered for this? So that
at least you can get the system back running."

Obviously, if such a process happened at all in Halifax, it happened
badly, as the multiple design failures---which one local engineer calls
"boneheaded"---demonstrate.

It's clear that such a process didn't work because the city placed
the design of the plant entirely in the hands of the private companies
whose interests laid in cutting costs, and not in providing a fail-safe
plant operational into the future.

A series of mechanical failures didn't doom the Halifax sewage
plant---the design/build contracting system did.

And yet, the city continues to use design/build contracting for
other large building projects, including the $40 million four-pad
hockey arena slated for Bedford [see correction below], the largest city project to be
undertaken since the sewage system was built.

It's unlikely that a poorly constructed hockey arena will result in
an environmental catastrophe, but it could very well lead to unexpected
future costs to taxpayers related to shoddy construction.

And if not the hockey arena, then so long as the city relies on
design/build contracting, some other project will fail.

It's only a matter of time.

Correction, 14 August: City staff informs me that contrary to what I wrote above, the four-pad arena is not a design/build contract but rather a design/build/operate contract. I was given incorrect information by a city councillor, but I should have double-checked. I regret the error.

However, for the record, I also don't favour design/build/operate projects. I think that either d/b/o OR the traditional method of contracting achieves the desired goal with regard to getting the best project built. BUT, there are, in my opinion, other non-construction-specific considerations; specifically, wage and salary issues. I think that once these sort of projects are completed, they should be operated by well-paid public employees, both because they'll do a better job than lower paid private employees, and because it's the right thing to do--- government facilities should have well-paid, unionized employees. That's ultimately a political argument, however, so I left it out of the article. —TB

Animation: How it Broke

Infographic: how it broke

Fig. 1

The plant operating properly: sewage from a deep tunnel enters a wet well 85 feet below the plant. Four pumps lift the sewage to the main floor of the plant, where the sewage is treated.

Fig. 2

A winter storm knocked out power to all the north end, including the Nova Scotia Power feed to the sewage plant. At this point everything worked as it was supposed to: the tunnel was closed off by a gate, and the plant was completely shut down.

Fig. 3

A technician turns on two backup generators and raises the gate.

Sewage flows into the wet well. But the load from the pumps is unevenly distributed between the generators. One overloads and turns itself off. The remaining working generator powers only one pump.

Fig 4

The single working pump can't handle the volume of sewage pouring into the plant. Sewage rises to the electrical junction boxes feeding the pumps, flows into the junction boxes, down the electrical conduits to the inside of the pump casings, shorting out the motors.

Comments
(30)

Remember, it's entirely possible to disagree without spiralling into a thread of negativity and personal attacks. We have the right to remove (and you have the right to report) any comments that go against our policy.