A few mistakes created many problems

Tacoma’s computer system project ran into trouble because of a few strategic mistakes, city employees and observers say.

Officials allowed it to grow too large. They refused outside oversight. And they failed to make sure employees were adequately trained. The management mistakes led to early meltdown, ongoing troubles and unexpected expenses that continue to this day.

Managers knew about many of the problems, but city records and interviews with people who worked on the project show they didn’t listen to critics who said it was too big and behind schedule.

They ignored members of a steering committee who called for an outside monitor.

They rejected a report that warned of problems a few months before the go-live date.

They pressed to meet a deadline even though they knew employees weren’t adequately trained to use the new system.

They exposed the city to a high level of risk by turning it on in one “big bang” rather than tackling the project in stages.

“This is a classic case” of poor implementation leading to problems, said Joshua Greenbaum, a Berkeley, Calif., consultant who follows large-scale computer systems and is familiar with the Tacoma project.

Managers defend their decisions, saying the computer system generally works fine and that the problems the city experienced since turning it on in October and November 2003 are normal bugs that should be expected in a project of this size.

Many of the problems aren’t a result of the computer not working correctly, they say, but rather employees not understanding how to make it work.

They say they saved taxpayers and utility ratepayers millions of dollars by turning it on all at once and not delaying it, and they insist it will make city government more efficient and eventually produce savings.

The project that officials bragged would put Tacoma on the cutting edge of e-government technology traces its roots to a more modest attempt to replace TPU’s customer service computer system.

That project – begun in the late 1990s during the scramble to prepare for Y2K – ran into trouble soon after it started, and the utility pulled the plug on it. By the time utility officials got back to shopping for a new customer service computer system, momentum was building within city government to buy a bigger, more comprehensive system that could replace computers in other departments, as well.

City and utilities officials, who historically had a rocky relationship, began talking about a plan that would link City Hall and city-owned utilities on one network.

Project leaders said their decision to replace more than 100 “legacy systems” – a piecemeal network of computers dating in some cases to the 1970s – with a single system was inevitable. They portrayed the city’s software as outdated, unreliable and headed for a crash.

They compared it with trying to keep an AMC Gremlin running and joked that some parts of the computer network were new when “Starsky & Hutch” was on television in the mid-1970s.

That might have been true of the teetering customer service and financial systems, but many other systems were working fine.

The 8-year-old PeopleSoft system, for example, was the industry standard for payroll and human resources information.

But in June 2002, following two years of meetings with consultants and vendors, a project team recommended replacing not only the two problematic networks – financial systems and utility customer service – but also human resources and work management, a sort of electronic in-basket that assigns tasks.

After looking at 28 proposals, the team picked TUI Consulting and SAP software over the doubts of some, including technology director David Otto and TPU customer service director Bill Schatz.

The price tag for the new system, as proposed: $50 million.

It’s a lot of money, officials said, but they believed it would be cheaper in the long run because they wouldn’t have to engineer an expensive way of linking different systems together, and it would eventually let them cut positions.

After SAP was chosen, the project grew larger.

Critics call it “scope creep,” jargon for a project that expands beyond its original scope and becomes hard to control.

Before it ended, the project included the tax and licensing department, the budget department and the building and land use department.

Ralph Johns, Tacoma’s deputy fire chief, said it became apparent that top managers wanted to use the new system for as much as they possibly could.

“The city knew they were making a huge financial investment,” Johns said. “They wanted to maximize it. They were swinging in the yard at anything they could swing at, hoping to hit an apple.”

The department had developed a good system for collecting data from its fire and medic calls, he said, and he worried the new system was too complicated and time-consuming. If it was too hard, he said firefighters would just stop entering their data.

The fire department also was concerned that private medical records would become widely available if they were linked to SAP.

The department succeeded in keeping its dispatch system separate from SAP.

Gary Pedersen, the now-retired director of the building and land use department, wasn’t successful at fighting off the system.

Pedersen said his department wasn’t originally supposed to switch over to SAP. It used a system called Tidemark that was installed in the mid-1990s and was considered the best for issuing building and land-use permits.

But he felt pressured to come into the fold and eventually gave up the fight.

“It’s difficult to explain,” Pedersen said. “Nobody would ever come out and say, ‘Do this or else.’ There was an undercurrent of ‘cooperate or get out of the way.’”

He still doesn’t know why managers were so eager to wrap his area into SAP.

“That’s always been a mystery to me,” Pedersen said. “Permits can be a stand-alone system.”

Project Manager Karen Larkin said she wanted permits on the new system to take advantage of efficiencies, not because it was better than Tidemark for handling permits. If the permit department weren’t linked to SAP, Larkin said, employees would have been forced to extract data from Tidemark and manually enter it into the other system.

Project leaders also say they based their decision to consolidate computer systems on advice from other governments – who said it would cost less in the long run – and after considerable discussion involving many people.

“Our philosophy was, ‘If SAP can do it, we would try to put it in,’” Larkin said.

Wired City

The bigger, more complicated solution appealed to the city’s desire to be viewed as a technology leader.

Since the heyday of the dot-com boom, Tacoma had marketed itself as “America’s No. 1 Wired City” in an attempt to lure high-tech companies, and the campaign influenced this new, more ambitious project.

News releases from the city, TUI Consulting and SAP frequently referred to Tacoma as the “Wired City.”

Officials bragged in news releases and speeches that Tacoma’s conversion to SAP software was the most complicated of its type in the nation and the second-most complicated in the world, behind one in Germany.

“Other cities and utilities have implemented larger systems in terms of customer base or population,” Larkin said in October 2003 when the first part of the system was turned on. “But Tacoma’s is by far the most diverse, serving a wide range of government services including police, fire, public works, finance and utilities.”

Tacoma’s project was “the buzz” at the June 2002 CIS conference, a computer system conference for utility industry customer service professionals, city employees noted later in a project meeting.

And project leaders were aware that the city was being watched by governments and computer industry observers around the world.

“These are things that will enhance our reputation as the nation’s most-wired city because we’ll be the first in the nation to do them,” Crisson told Nation’s Cities Weekly in an article published June 9, 2003.

The city found a like-minded partner to help with the undertaking.

After looking at 28 possible vendors in the competitive world of information technology, Tacoma selected TUI Consulting, an ambitious company based in Melbourne, Australia.

By the time the City Council voted to award TUI a $26 million contract, the decision to go with a unified system using SAP software already had been made by city staff. (TUI eventually wound up getting $29.6 million in the deal.)

“Tacoma once again is on the leading edge,” City Manager Jim Walton declared when the system was turned on.

No outside oversight

Getting to the leading edge required Tacoma and TUI to enter uncharted territory. Never before had a city merged the computer network of a multiservice utility with its general government.

The city chose to use SAP software, a powerful yet complex and unforgiving system that required extensive user training and a new way of doing business. And the city elected to switch over to the system in just two main phases within about a month – one for general government and one for the utilities – rather than slowly phase it in.

Despite the risks, the city failed to take oversight precautions that many experts recommend.

“In a project of this size, you would expect to have an external services firm as the project manager,” said Jim Shepherd, senior vice president of AMR Research, a Boston company that provides strategic advice to executives with a focus on technology.

Instead, the city used a hybrid approach with Karen Larkin, who normally worked as assistant public works director, as an internal manager and TUI founder Ballu Khan as a co-project manager.

A third party also might have been brought in to make sure the project was on track.

“It’s not uncommon to hire a third party to do periodic audits,” Shepherd said. “That would generally catch major problems.”

But top managers consistently rebuffed the idea of doing a third-party audit, and as the project moved closer to completion, disputes arose over the way it was managed, according to e-mails, city records and interviews.

The city did create a nine-member steering committee of department heads and other managers to help oversee the project. But the record shows their concerns were often ignored.

Nearly a year before the go-live dates, members of the steering committee called for an independent quality assurance study to make sure the project was going as planned.

Schatz, the TPU customer services manager, asked about bringing in a third party to check on the project as far back as September 2002, steering committee minutes show.

TUI boss Khan, who didn’t think an outside monitor was necessary, noted that SAP is “very tough on quality checks,” meaning it was known for conducting a thorough status check for customers before the system went live.

The steering committee continued to press for an independent quality assurance report. And in March 2003, with seven months remaining until the go-live dates, the members sent a memo to Crisson and Ray Corpuz, then city manager, outlining several concerns, including:

• Multiple changes in the TUI contract: “The TUI contracts have been modified to the point that the committee has lost the ability to track progress,” the committee wrote. Among the reasons for the contract changes were a delay in the software arriving from SAP and a change in the way the city bought computer hardware for the project.

• A shortened time schedule: “Compressing” the schedule threatened the ability to properly test the system, the committee said.

• Lack of a detailed project plan: Steering committee members said TUI was selected with the perception that it was weak in “project management skills” and that it did not have experience managing a project like Tacoma’s. Those weaknesses, combined with the changes in the contract – and its growing size – raised concern among the committee members about the lack of a detailed project plan.

Steering committee members, worried about the fate of the project, called for an independent assessment saying it was “critical to the success of this project” and “a prudent and customary practice for a project of this magnitude.”

But Larkin and her supervisors, Crisson and Corpuz – and later Walton – held firm that an outside check-up wasn’t needed, so one was never completed. Khan agreed with the decision.

“We talked about it a number of times and frankly I was never persuaded that it would have added a lot of value to the project,” Crisson said.

“We didn’t need someone to come and relook at all that decision-making,” Larkin said.

“It would’ve been a total waste of money,” Khan said. “How can a review team come and dig through everything and try to poke holes in it or say it’s great in a matter of a week? It’s a physical impossibility. It’s nonsensical.”

Instead of allowing an outsider to assess the project, managers elected to keep the work in-house by appointing two city officials – Tacoma Water Superintendent Ken Merry and public works director Bill Pugh – and two executives from TUI Consulting to conduct an internal quality check.

They generally said the project was on track, but failed to interview members of the steering committee for their report, something they later admitted was an oversight.

Schatz, the TPU customer services manager and a steering committee member, called it a “kangaroo court.”

After the internal report was completed, Larkin began pushing to “sunset” the steering committee and replace it with a larger group called the Leadership Forum. As the project grew closer to completion, it became more important to bring in people from every city department, not just a handful of top officials, Larkin said.

But the members of the steering committee viewed it as an attempt to eliminate the one group that tried to provide critical oversight of the project. And they didn’t believe the timing was a coincidence.

“Twice we asked for a third-party review and not long after that we were informed we were being sunsetted,” said Johns, the deputy fire chief.

Corpuz and Crisson changed their minds about eliminating the group, but only after the members objected. Even so, some members complained the steering committee wasn’t able to challenge project leaders.

The committee was established to provide executive oversight of the project, but in fact it ended up serving primarily as an audience for Microsoft PowerPoint presentations from Larkin, said finance director Steve Marcotte. The presentations, he said, assured them that “everything was on track.”

Crisson said he thought they had a good arrangement, though the opinion wasn’t unanimous.

Warning sign rejected

Even without the independent quality assurance check that steering committee members wanted, top managers knew they had problems by summer 2003.

With a few months to go before the go-live date, SAP sent a team to Tacoma to look at the project, a standard practice.

The team concluded the Tacoma project had “serious issues” facing it. And even though the SAP team believed the city could go live with the new system, it warned it was likely to experience “numerous problems” unless those issues were quickly resolved.

Several aspects of the project were classified “high risk,” the assessment shows, including the project plan and training.

Both go-live dates were jeopardized by the problems with the project plan, the report showed.

“We feel that the project is behind schedule,” the SAP team wrote. “And because project plans have not been adequately maintained, we cannot determine a revised timeline.”

Rather than raising concern among project leaders, managers rejected the findings and refused to pay for the report.

It contained so many factual errors, Larkin said, it showed that the SAP employees who prepared it didn’t understand the city’s basic business needs.

“We had city folks who understood SAP better than SAP does,” she said.

After the city received an invoice for the report, Larkin wrote to SAP America executives, reiterating the city’s position that it was a worthless report and the city would not pay for it.

“As a result of the factual errors and demonstrated lack of understanding of the City’s business needs … we rejected the review as useless,” Larkin wrote.

No members of the steering committee saw the critical report.

No members of the leadership forum saw it.

No City Council members saw it.

The News Tribune made several requests for the report under the Washington open records act. After several weeks and several requests, city officials finally determined what it was and where to find a copy of it.

Looking back, Walton said the report should have generated more concern.

“In hindsight, yeah,” Walton said. “In the moment, we reached a different conclusion.”

But Crisson believes the city made the right decision.

“The gist of the findings were that we weren’t getting enough people to training,” Crisson said. “It didn’t contain anything we didn’t already know. I wasn’t interested in paying for it.”

Inadequate training

Even though city officials knew from their research choosing the system that training was crucial, they failed to make sure employees were ready to use the system when it was turned on.

It was important, they learned, not only to teach people how to use the computer system, but also to prepare them for the notion that the SAP software would completely change the way many employees did their jobs.

Yet as managers rushed to prepare for the go-live dates in October and November 2003, Gartner Inc., a company hired by the city to survey employee readiness for the system, identified training as a “major concern.”

Most of the training classes were held during the summer when employees were away on vacation, and those who weren’t away often said they were too busy with their jobs to attend the classes. Some who did attend classes complained that they were cursory.

He complained in an April 2003 e-mail to Schatz and the other steering committee members that city and TUI employees scheduled “required” training events without taking into account the complexities of police shifts and overtime implications.

“At this time, we have directed our personnel to disregard” training meeting invitations, Mielbrecht wrote.

The David Brame scandal is another reason managers didn’t do enough to ensure their employees attended training classes, Khan said.

People became distracted after the city’s police chief fatally shot his wife and himself, touching off investigations throughout City Hall and subjecting the city to the glare of national media attention, he said.

“It took up a lot of energy,” Khan said.

In the final weeks, officials installed digital countdown clocks at City Hall and the TPU administration building to show how much time remained – down to the second – until the go-live moment.

With less than a month until the City Hall clock struck zero, Gartner officials said Tacoma was still experiencing “above average risk” because of training issues.

There were other concerns besides training, and some employees called for a delay in the go-live date.

“STOP THE CLOCK!” one employee wrote in an e-mail to project managers warning that there was too much work remaining to be done.

But Crisson, Walton and Larkin pressed ahead, saying they had the support of weary computer workers who wanted to finish the project sooner rather than take a break.

They considered a delay in the go-live date but calculated it would cost $6 million to $7 million to delay it three months. The extra cost is largely what it would cost to keep the consultants working on the project beyond the original target date. The way the contract was written, if the city requested the delay it would bear the cost of it.

In hindsight, Crisson and Larkin admit they could have done a better job training employees.

In August 2004, nearly a year after the go-live date, one-third of the 890 employees who answered a customer service survey said they still didn’t have enough training to get their daily work done.

“We probably could have benefited from a little more training, but there would have been a cost to that if it took more time to go live,” Crisson said. “To our own employees and ourselves, we probably oversold the system a little bit in terms of how easy this would be.”

Never miss a local story.

Sign up today for unlimited digital access to our website, apps, the digital newspaper and more.