State Representative from Washington's 36th Legislative District

A $300 million mistake

July 21, 2009

Dep't of Information Services New Data Center

In running for the Legislature, I enthusiastically embraced Barack Obama’s vision for transparency in government. I’m striving for openness, candor and honesty in my public dialogue with the 130,000 people of our district and I’m doing all I can to embrace a new type of politics. Yet the incentives internally, within the political subculture, all go in the opposite direction. If you want to get ahead then keep your head down, listen and learn, take incremental steps, and work your way up by keeping out of trouble. Pick and choose your battles. If you can’t win, let it go.

Great in theory, tough in reality. It’s good advice that I may already have failed.

Yesterday, Rep. Hans Dunshee, an independent thinker if ever there was one, and I sent a letter to Governor Gregoire expressing our reservation and opposition to the state Department of Information Services‘ new $300 million data center in Olympia. We asked her to secure an independent second opinion before allowing the sale of $300 million in bonds for a project that is, in my view, bad technology and bad finances.

The institutional infrastructure of bureaucracy in Olympia is enthralled with this plan–a modern, new data center consolidating their old systems blocks from the Capitol Campus. For years the agency has been lobbying the Legislature. After years, with passion and support from the Olympia community and business and labor interests who would gain from construction and operations, they pushed it over the top. I understand their position and respect their success. Also, I am not claiming some blanket statement that the agency doesn’t do some good work and deliver value to the public.

But the agency’s analysis of their idea and proposal is shocking in its lack of quality research, new data, national context or insight into technology trends. The decision to build a state-owned and operated data center when all of the major technology trends are going toward the cloud. As far as I can tell, the Information Services Board, the board overseeing the agency, didn’t even approve the proposal…it was so well accepted. Or perhaps not. I don’t claim to know the answer.

I believe we will deeply regret this decision in the long-term. It is bad technology and bad strategy and bad finances. A fraction of this amount invested in real-time, interactive, citizen-friendly applications and services would revolutionize how our students learn, how our foster kids keep track of records, and how we reduce costs in health care.

Of course, I readily acknowledge and accept that I was too late to the table and too far on the back bench to effect the outcome of this intense lobbying effort by the bureaucracy. Hans and some other technology and investment oriented legislators tried but the forces advocating the new data center had real strength. They also hid behind the technical reasoning. Good and smart folks can and do disagree technically and financially, and there is certainly a real chance I’m dead wrong on this one, but my personal and professional view is that this is an excuse for a long-term jobs program for the IT bureaucracy.

Perhaps it is bad politics to point out that the Democratically-controlled Legislature and Governor have gone along after years of lobbying by the agency with a decision that is really only in the agency’s own best interests, but in my view it’s the truth of the matter.

We are going through the most dramatic structural shift in technology and government in history. Now is the time to move toward a new strategy and new approach around the cloud, virtualization, applications, mobile and the Internet, not invest $300 million in a huge new IT building just blocks from the Capitol. This investment will not improve services for citizens. That’s tough to swallow.

If I can’t be open, transparent and honest about a decision that I believe is a waste of $300 million, why did I run for office? If not now, when? This, it seems to me, is my job as a citizen legislator.

Recently we spent considerable time reviewing the financial and technical materials provided by the Department
of Information Services to make the case for the state’s new data center.

Upon review of the material, we are deeply troubled by the weakness of the technical and financial support
behind this decision, and fear the state is potentially making a $300 million mistake that will haunt us for decades
to come. The lack of an independent, quality, detailed, technically-oriented business case is deeply troubling. We
strongly encourage you to formally review this decision before bonds are sold in the coming days.

There are essentially four main strategic options for the state.

1. Utilize cloud services from commercial providers such as Google, Microsoft, Amazon or others as many
companies and governments are doing today.
2. Purchase state-owned computing equipment, but locate it in one or more commercial data centers, such as
those located throughout the state by a number of providers.
3. Build a state owned data center, but locate it somewhere other than Olympia—a place where total cost of
ownership including land, energy, labor and infrastructure is less expensive and where geographic diversity
provides system reliability.
4. Build a state owned and operated data center on the Capitol Campus.
A compelling case can be made that the DIS staff decision—which appears to have been made without formal
ISB approval or independent financial analysis—to select number four is seriously flawed. We acknowledge the
years of hard work, discussion and review and do not mean to imply that there is not great thought behind this
decision. Yet tactically a vast majority of the financial and technical analysis is limited to options 3 and 4
whereas the larger technology strategy question—and the public policy issues—are driven by options 1 and 2.

We also respect and, in fact, fully appreciate that option 4 is unquestionably in the best interests of the current
agency since employees wish to retain IT jobs and remain in Olympia in a modern, consolidated, conveniently-
located building.

The DIS business case fails to seriously explore the larger strategic question facing government technology today:
How best to efficiently and effectively move away from hardware-centric, expensive, proprietary, silos of data
trapped in old databases to open, transparent, flexible, accessible, customer-oriented applications available via the
Internet? This massive investment of public dollars will not materially improve public services for citizens. It is
unlikely to substantially save costs for taxpayers. It will not ensure a higher quality disaster recovery response.

Public sector IT experts predict that within just a few years up to 50% of government agencies nationwide will
outsource most data to the cloud. Washington is home to many of the leading providers of this rapidly evolving
commodity service where improved security, disaster recovery and lower costs are being driven by almost
universal adoption by both the public and private sectors. Still, our own state government has yet to move in this
direction in any material way. We are troubled that option 1 is covered in just over one page in the agency’s
analysis and is dismissed outright without technical or financial support.

We applaud you for hiring Tony Tortorice. He has been gracious and responsive and we are impressed with his
conviction for his new role and for this monumental project. He clearly has deep and impressive technical
knowledge. We look forward to learning from him for many years to come. However, given the magnitude of the
cost to taxpayers, the lack of an independent, technical, second opinion and the lack of comparative data for all of
the options, we encourage you in the strongest way possible to postpone the imminent sale of the bonds and
review this decision. We encourage you to seek outside counsel on this major decision.

We are speaking out because we feel a fiduciary obligation to let you know, directly and personally, that we
believe this decision is fundamentally inconsistent with the technical, financial and strategic direction of public
and private sector innovators worldwide. We believe this decision, if allowed to stand, will one day be deeply
regretted.

Does the cloud have some potential benefits? Sure. But like every miracle touted IT technology, it’s not going to solve every problem and slice bread more cleanly. Right now – today – it’s still immature, a bunch of vendor promisses and pretty marketing PowerPoints about what will “be in the next release, real soon now”. So while there are some benefits, you are not going to gain every bit of the hype in cold $ savings – and you take on a lot of risk.

It’s worth asking the question – but it is not a step to be taken lightly – beyond the same security risks you have with any data center, there are new risks and a lot of unknowns.

A substantial part of the state infrastructure that I’m aware of is moving to virtualized machines. “Moving to the cloud” is often seen as equivalent, but it’s not. These are two interrelated but different steps.

Purely in the virtualization world, the tools and technologies are immature. If it’s all in-house, you can manage it, but you lose a certain level of information about the innards of the network. That’s a risk tradeoff that has proven acceptable because of the benefits of virtualization.

For example – most data centers deploy a set of tools, services and equipment to protect and monitor traffic. These tools – Intrusion Protection Services (IPS), Firewalls, etc. – are crtical to the security of the data center. The virtual network traffic is invisible to these tools. There are work-arounds (what you do is create multiple networks, run the traffic OUT of the virtual world into a physical switch you can connect your monitoring tools to and then run it back into the virtual world).

BUT:

Taking the next step and moving those processes to a public utility model (“moving to the cloud”) exposes you to these risks, doesn’t allow you to deploy the work-arounds and adds even MORE risk. Why? Because there are now other peoples processes running along side yours – and you can’t see what they are doing.

You would not just let anybody walk into the data center and plug their laptop into the network. You wouldn’t let the execute systems administration tasks on your computers.

With a cloud model, you can’t prevent it…

So you contractually require all your processes run on an isolated set of machines, on a separate network. And then all you’ve really done is outsource the ownership and maintenance of the data center.

Does that offer savings, scalability and benefits … depends on a lot of things.

Your analysis is strong, of course, and I reiterate my point that the cloud isn’t the automatic answer. Yet the question, analysis, study and review is essential. The agency’s business plan covered the entire scope of every aspect of the functionality in one and a half pages. It’s just not robust enough on these tough issues.

I can’t get a handle on how actively the state has embraced true virtualization models and systems, so that’s something I need to learn from DIS more thoroughly.

Since this is state government, it would be very safe to assume that there are more things in play here than just construction interests. When you consider the fact that most government organizations run legacy applications (i.e. mainframe and as/400 based applications) that are absolutely critical to operations, it would not be a viable option to use the cloud. To the best of my knowledge no current cloud provider offers anything other than unix based OS’s and some offer Windows based OS’s.

The knee jerk reaction to this would be “rewrite the applications!” that wouldn’t be the most prudent thing either, as that takes a lot of time, and depending on how many applications could cost well over the $300MM earmarked for a datacenter. Rewrites open you up to endless headaches from contracting organizations that don’t fully understand what it is they’re rewriting. Introducing bugs, downtime, blame games and subsequently causing heads to roll.

I might also add that I have not yet heard of any major corporation wholesale (or even piecemeal for that matter) moving their entire operations to the cloud. It’s just not something that is going to happen any time soon, no one wants to trust the backbone of their business to anyone whose sole purpose is not (i.e. dedicated staff) is to ensure that the interests of their business are protected.

Why would the state build a data center when the private sector can provide data center hosting as a service to the state? This is what many private sector companies do. It is available now, it works, increases availability, increases security, can address disaster recovery issues, reduces facilities and personnel costs. If the state wants to operate more like a business, then it needs to look at what the private sector is doing, and adopt those practices.

Additionally why not seriously consider moving agencies to open source solutions, for example Google Docs which would provide word processing, spreadsheets and presentations for significantly lower cost than Microsoft. Additionally email, shared calendaring, web sites, blogs and many other services are available.

The legislature needs to require state agencies to use open source solutions, or provide justification as to why open source would not work before purchasing commercial software.

Why is the State Department of Information Services (DIS) brokering telecommunications services to agencies at a marked up price? DIS is acting as a middle man to itself … this does not make any sense.

Why Olympia? If the State of Washington needs what amounts to a server farm, why build it on expensive land in an earthquake zone when you have the whole eastern part of the state to use instead? Put it in Ritzville and run a wire to Olympia.