When Darius Jackson became ING’s head of IT infrastructure support and service delivery in January 2005, his job was to clean up a mess. Two years earlier, the financial services company had outsourced its IT infrastructure (hardware, software, help desk and so on) to a major service provider in a seven-year, $600 million deal. But now the business leaders of the company are worried that they aren’t getting the value they want out of the relationship.

Jackson has a crowbar for leveraging more value out of the deal, though he hasn’t used it.

It’s called the benchmarking clause, a small paragraph hidden in a vast, complex outsourcing contract that gives him the right to assess the outsourcer’s prices using an independent benchmarking firm. If those prices turn out to be way above the going rate, he could bring his outsourcer back to the negotiating table. The outsourcer probably wouldn’t be too pleased with Jackson should he choose to exercise that right. "They like to look at the relationship as a partnership," says Jackson. "Their preference is that you come and talk to them about issues and concerns and try to work through them as opposed to looking under the covers."

But what Jackson and other IT leaders may not know is that the power of the benchmarking clause, which is hidden inside most outsourcing agreements today, has diminished and the practice of price benchmarking is in danger of disappearing altogether. "Suppliers have scant enthusiasm for benchmarking, which shaves their margins and tends to be invoked just as their contracts start to become profitable," says George Kimball, a partner with law firm Baker & McKenzie who represents outsourcing customers. Not surprisingly, benchmarking clauses are among the most contentious and negotiated terms in an outsourcing contract, says Kimball.

In fact, the major IT service providers have launched an all-out assault on benchmarking, fighting to turn it to their advantage. "[They are restricting] what you can benchmark, how often you can benchmark and the amount the benchmark can reduce the price of services," says Mark Robinson, executive director of advisory services sourcing consultancy EquaTerra.

Other experts say the big outsourcers are looking to avoid the process altogether. "They want to stonewall it," says Howard Rubin, a senior adviser to research company Gartner.

All of the major providers we spoke to for this story—EDS, Hewlett-Packard and IBM—deny that they are out to kill benchmarking. However, all express frustration with the current state of the practice and all are trying to change it—to their benefit. "Benchmarking isn’t ever going to go away, but I think it will change," says Jon Stewart, VP of market management for Electronic Data Systems (EDS). "I think we need, first and foremost, more credible [benchmarking firms]. There’s not a lot of players in that space. Second, there needs to be some recognition of what you can realistically expect from a benchmark. The old mind-set is, ‘Ah, I’ll go to a benchmarker and they will have a robust view of the market.’ They don’t."

If the power of the benchmarking clause diminishes, CIOs will concede one of the only tools for controlling the cost of outsourced services. The risk with any outsourcing contract is that you end up paying through the nose for services that should be getting cheaper, particularly in the infrastructure area, where prices for hardware are dropping constantly. "If you leave it up to the vendor, there won’t be any benchmarking. Your next shot to adjust prices is when the deal’s up," says Robert Finkel, partner with law firm Milbank, Tweed, Hadley & McCloy.

The Birth of the Benchmarking Clause

The benchmarking clause dates back to the mid-’90s, when the number of mega-outsourcing deals began to explode, along with the lengths of the agreements. Signing a 10-year deal with a multibillion-dollar price tag was a big risk—one that customers wanted to mitigate.

In the beginning, says Kimball, benchmarking clauses were kinder and gentler—designed only to bring outsourcer and customer together to confer diplomatically if prices seemed higher than market averages. But as the decade drew near a close, CIOs and their business colleagues got nervous. They wanted some assurance that increasing competition among the outsourcers and constantly dropping costs of computing power would be reflected in what they were paying. Outsourcing consultants and external counsel were only too happy to oblige. Midwifed by such advisers, new benchmarking clauses emerged. With teeth.

The clauses not only enabled periodic benchmarking, they forced a remedy. Should charges fail to fall within the bottom 10 percent of the market, the supplier had to lower its prices. Some more aggressive clauses even required retro¿active reductions. "They were planned to be punitive to the providers," says Rubin.

In the vendor feeding frenzy of the time, outsourcers signed these customer-centric provisions left and right. "Back then, it was all green pastures and room enough for everyone. Outsourcers just chased deals," says Stewart. "There wasn’t a lot of thinking about how the market might change in the future."

A few years later, however, when some large customers began to flex their benchmarking muscles, outsourcers felt the pain. In one of the most extreme cases in 2002, Britain’s Cable & Wireless sued IBM for more than $200 million for price gouging after a disputed benchmark. IBM counter¿sued and brought Cable and Wireless’s benchmarking company, Compass, into the lawsuit. They ultimately settled out of court. The outsourcers suddenly woke up, as if from a nightmare. "They started to say, What the hell have we agreed to?" says Geraldine Fox, global sourcing practice lead for Compass. "The price standard they were being held to—the lowest decile—was ridiculously strict."

Why Vendors Hate It

Attribute the sharpness of the bite to the emergence of benchmarking companies around 1998. Benchmarkers had been evaluating IT shops’ internal operations since the late ’80s, but when the outsourcing boom hit they came up with an enticing pitch for CIOs: Let us invoke the benchmarking clause, and we guarantee to cut your prices. With benchmarks costing anywhere from $100,000 to a million dollars, depending upon scope, it was hard for benchmarking firms to seal the deal without pledging to get at least some of that money back. But that promise poisoned their objectivity, some say. "When you say we guarantee to save you 20 percent, you’re not really being objective," says Adam Strichman, senior partner at research and benchmarking company Nautilus Advisors and former director of outsourcing strategies at Meta Group.

The outsourcers screamed foul. The whole benchmarking process, they contended, was a crock: Service provider cost models are complicated; there are financial and operational dependencies among different services; outsourced services are rarely commodities but rather are sold in varying combinations upon a wide array of terms. For example, the price for mainframe services can vary by plus or minus 40 percent, says Stewart of EDS, depending on what software is running on the machines. "You have to understand the client environment, service levels and the financial assumptions that went into the deal to come up with an accurate benchmark," he says. "And benchmarkers don’t know the deal specifics."

Furthermore, say the outsourcers, the benchmarkers’ attempts to account for the variations among companies—what is euphemistically known as "normalization" in the industry—isn’t an adequate substitute for real numbers and knowledge. "The vendors complain benchmarking is more art than science," says Robinson of EquaTerra. "And a black art at that."

The Benchmarking Business

Benchmarkers, whose overall market now exceeds $200 million per year, according to one industry analyst who asked not to be named, concede some of these points. "There is an art to it," admits Strichman. "Benchmarking is far more than statistical analysis and number-crunching." Benchmarkers say that over the years they have refined their processes to reconcile unlike data and adjust for differences such as the client environment and service levels.

But while benchmarking may require some apples-to-oranges comparisons and fact-finding to adjust for different environments, it’s the only method currently available for CIOs to ensure the competitiveness of the prices they’re paying. "There’s value in the benchmarking clause. It gives a customer security when signing a long-term contract with a single provider," says Neil Barton, Hewlett-Packard Services’ benchmarking manager for Europe.

And while outsourcers complain that the benchmarkers’ numbers don’t add up, they are loath to admit that their own numbers don’t add up either. Service providers typically grant customers a great price on IT services on day one but backload their costs to recover initial investments later on. Compass reports seeing year-one savings of as much as 18 percent (usually 10 percent to 15 percent) turn into costs in excess of 23 percent above market rate by year three and more than 35 percent in longer deals. Outsourcers aren’t likely to open up their books and show you how they’ve arrived at your charges, so benchmarking is an important tool. "Whatever its limitations," says Kimball, "benchmarking can be an effective catalyst for renegotiations that raise service levels and reduce charges."

The Vendors Strike Back

A few years ago, major outsourcers began taking active steps to restrict benchmarking. They created dedicated teams to analyze costs and negotiate benchmarking clauses to their advantage. "What was once just grumbling turned into organized grumbling," says Strichman. "Now every major service provider has a group that manages the process." These groups are charged with analyzing internal intelligence; benchmarking to calibrate their pricing models (the outsourcers use the benchmarkers’ services too); and managing the process when a customer decides to invoke its benchmarking right.

Alan Yamamoto set up such a group at IBM five years ago. True to Big Blue form, the company has filed a patent in this area (see "Patent (Fight) Pending," this page). Meanwhile, IBM’s arch rival, HP, has created a team, as has EDS. "[Vendors] definitely have gotten more vigorous in protecting their interests," says Stuart Harris, a partner with outsourcing consultancy TPI. "I can’t say they’re systematically obstructive, but some of their clients might."

One of the tactics these teams use is to make a benchmarking clause so specific in its requirements that it becomes too difficult or expensive to invoke—for example, requiring too many peers in the benchmarking group (five is generally plenty, say benchmarking companies). Other terms the outsourcers may seek include negotiating a detailed limit on how soon or how often the customer can benchmark, the opportunity to review and negotiate draft findings, and caps on mandatory reductions in charges, among others.

Shutting Off the Data

But perhaps the most obstructive effort by outsourcers so far is their attempt to stymie benchmarker data gathering. In order to build their market cost estimates, benchmarkers need to pool data from many different outsourcing customers. Since the beginning of the benchmarking era, outsourcers have allowed benchmarkers to reuse the data they gather during their benchmarking engagements as long as they agree not to reveal customer names.

Yet within the past year and a half, vendors have begun asking benchmarkers to sign a legally binding document promising that they will not reuse the data they gather from the outsourcer’s customers, thereby preventing the benchmarkers from making comparisons across companies—the very essence of what they do and the foundation for the service they provide their customers. Indeed, some providers, such as IBM and EDS, have banned data reuse—with rare exceptions for particularly large and determined customers. This is despite the fact that most service providers use third-party benchmarkers themselves to construct and maintain competitive deals. "If one believes in the reuse of pricing data, you have some obligation to permit that to happen," says HP’s Barton, noting that HP employs the services of benchmarking companies Compass, Gartner and Germany-based Maturity Consulting. HP is in the process of "reviewing" its data reuse policy, he says.

EDS’s Stewart sees the contradiction: He complains that the benchmarkers lack sufficient data to make good pricing estimates, and now he’s exacerbating the problem. He doesn’t much care. "The reality is I have better data than the benchmarkers do," Stewart says. "We participated in 9,000 deals last year. We have much more robust information. We don’t need [benchmarkers] to define our view of the market."