Business Intelligence projects are famous for low success rates, high costs and time overruns. The economics of BI are visibly broken, and have been for years. Yet BI remains the #1 technology priority according to Gartner. We could paraphrase Lee Iacocca and say: People want economical Business Intelligence solutions and they will pay ANY price to get it.

Nobody argues with the need for more Business Intelligence; BI is one of the few remaining IT initiatives that can make companies more competitive. But only the largest companies can live with the costs or the high failure rates. BI is a luxury.

I believe that the bad economics of BI are rooted in the IT department/BI vendor duopoly on BI infrastructure. This post focuses on IT’s inability to deliver efficient BI projects; I will write about the BI industry in my next blog:

There are three fundamental reasons why IT departments in their current form fail to deliver economical BI solutions:

1) They don’t understand elastic scale

IT departments are good at scaling: adding more and more hardware and software but scaling makes sense for tasks that are highly predictable. Given the ad hoc nature of BI we not only need to increase the compute power when we need it for a complex queries but we also need to be able to decrease the compute power when it’s not needed to keep the costs down. Elastic is more important than scalable. And this precisely why internal BI solutions will always be either too expensive or too slow for complex queries…

2) They try to control BI with a single version of the truth

While the volatility of business environment is increasing the IT departments are trying to button up the business knowledge (data, metadata, processes) into a top-down, inflexible and lengthy process that should produce a single version of truth. The problem is that the underlying business is changing so rapidly that by the time this is done the resulting analysis and reports are not correct anymore and the BI project becomes shelfware.

3) They cannot measure success of BI

“If you can’t measure it, it’s not worth doing!” is one of the selling point of BI but it is difficult to measure the success of BI projects. IT delivers on initiatives that are quantifiable (throughput, response time, performance, data sizes) and since the data size is one of the few easily measured aspects of BI it is the only metric where IT can claim success. This is why we often read about terabyte and petabyte datawarehouses. But it is a small portion of the BI market (2%) and they happen to be places where data goes to die.

Share this:

My plan for today was to write more about the “Business-IT chasm” but I came across a great blog post written by Jorge Camoes that reveals the business perspective of this divide. There is nothing better than the first hand experience:

IT will try to change your project, naturally. Try to avoid the “security bomb” (their favorite). You know how poor their expensive BI toys are, and you should know what they can and can’t do with them. Minor concessions can earn you some points. When they tell you they can’t implement your core ideas be prepared to fake genuine surprise, compare costs (again) and emphatically say that their options clearly don’t meet the organization’s needs.

My suspicion that business has limited ability to influence the electrical engineers is demonstrated by this quote:

Pissing off the IT department is one of the most enjoyable games in corporate life, but be a gentleman and don’t make them look stupid. They don’t usually have a good sense of humour and take their quest to conquer the world very seriously. If you really want to implement the dashboard, don’t make it an island if you can avoid it (connect it to the tables in the IT infrastructure, instead of copy/pasting data).

Share this:

I believe that readers of my blog are familiar with the chart below. It is the classical “Crossing the chasm” diagram from Geoffrey A. Moore’s book Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers. This book was published back in 1991 and it is still number two on “Twelve Business Books in One Hour for the Busy CEO” list. The main argument that Geoffrey Moore makes here is that the “early adopters” have no ability to influence “early majority” and it leads to a chasm that is very difficult for startups (or any company with disruptive technology) to cross:

This book was written in the days when startups typically sold to electrical engineers (a.k.a. IT) and Brad Burnham described it well in his recent post:

In the old days, electrical engineers focused on getting computers to work not on getting people to engage with the systems built on top of those computers. The folks that built enterprise software were vaguely aware that their systems had to be accessible to the humans that used them but they had a huge advantage. The people who used them did so as part of their job, they were trained to use them and fired if they could not figure them out.

This is why even Wikipedia uses the following example to describe the end-user:

The end-user or consumer may differ from the person who purchases the product. For instance, a zookeeper, the customer, might purchase elephant food for an end-user: the elephant.

But virtually no startup gets funded today if it sells directly to electrical engineers. Innovation happens in the consumer space anyway and so the assumption is that any disruptive technology (social networks, SaaS, web2.0…) gets adopted first by the end-users and then it is picked up by the IT. But here comes the new chasm: low ability of end-users to influence IT:

This chasm is the new manifestation of the classical “Business-IT” gap but this time is the innovation flow reversed: business leads and IT follows. And this new flow doesn’t make it any easier to cross the new chasm…