Database myths and legends (Part 3) BI (Business Intelligence) is about extracting information from data and data mining is an important part of that process. Data mining is a process that looks for patterns in data, so in a sense it is like querying the data. The crucial differences between simply querying the data and data mining can be summed up as intent and scale.

When humans query data we start with an idea, such as: "I think that we sell more DVDs to males than to females." And then we run a query to test the idea and the answer either confirms or disproves our hypothesis. A data mining algorithm doesn't have ideas. It has no intention of testing ideas for the simple reason that it doesn't have any.

What it does have is huge processing power at its disposal, so it simply tests a very large number of possible correlations. This can be done by firing a very large number of queries at the database, but that approach can be very slow.

The algorithms used nowadays often use more sophisticated approaches; they can, for example, create a multi-dimensional data structure and then examine it looking for patterns, and/or outliers. When they find something of interest, they flag it for attention.

It is easier to illustrate the difference between querying and data mining with a good example and, already firmly enshrined in BI mythology, is the "beer and diapers" story.

It goes (with minor variations) like this:

Some time ago, Wal-Mart decided to combine the data from its loyalty card system with that from its point of sale systems. The former provided Wal-Mart with demographic data about its customers, the latter told it where, when and what those customers bought. Once combined, the data was mined extensively and many correlations appeared. Some of these were obvious; people who buy gin are also likely to buy tonic. They often also buy lemons. However, one correlation stood out like a sore thumb because it was so unexpected.

On Friday afternoons, young American males who buy diapers (nappies) also have a predisposition to buy beer. No one had predicted that result, so no one would ever have even asked the question in the first place. Hence, this is an excellent example of the difference between data mining and querying.

The story goes on that, once the correlation was uncovered, it was easy to back extrapolate from the effect to the cause.

Young American males frequently indulge in ritualised carousing behaviour with friends of Friday nights.

Carousing usually involves the consumption of beer.

Most young American males only buy diapers after they have fathered offspring.

Offspring acquisition is a known carousing inhibitor.

So the proud new father is walking around the store on Friday afternoon. He knows there is no way that he is going to get out of the house to join his mates at the bar. However, there is nothing to stop him from drinking beer at home. All he needs is to be reminded of that fact. After seeing the results of the data mining, Wal-Mart moved the beer next to the diapers and beer sales went up.

Like all good myths, the beer and diapers story does have its origins in fact; but sadly, almost all of the detail (and specifically the detail that makes it a great BI story) is probably fabrication. We are all indebted to Daniel Power for uncovering the origins of the story. He provides all of the detail here.

In short, he traced the story back much further than I would have believed, way back to 1992. At that time, Thomas Blischok was the manager of a group at Teradata. His group looked at point-of-sale data from Osco Drug stores - 1.2 million baskets worth in all. It isn't clear what tools the team was using. They are described as "state-of-the-art, query generation tools", which were undoubtedly leading edge in 1992.

Those queries revealed that, between 5pm and 7pm, customers tended to co-purchase beer and diapers. No correlation with age or gender was established; although it isn't clear whether these questions were ever asked.

In addition, the store chain does not appear to have exploited the information by moving the products around. So we have a correlation between beer, diapers and time, but no correlations with age, gender or day. Worst of all, it is very much open to question whether the techniques used would qualify as data mining in its modern sense.

So, where does all that leave our beer and diapers story as an example of data mining? Well, I for one am prepared to accept that it didn't really happen in the way my grandfather described it to me as I sat upon his knee. Sigh.

However, that doesn't mean that we have to consign the story to Room 101 (for those readers outside the UK, also a popular TV comedy). The story is so popular because it is a good illustration of the difference between querying and data mining. The facts don't change that at all.

The image of a forlorn young man trudging round a supermarket on Friday night; his day suddenly brightened by the sight of a stack of beer sitting incongruously next to a pile of diapers is somehow wonderfully compelling. And none of us would have asked that question in the first place.

There is no reason we cannot continue to use it as an illustrative story; as long as we are aware that it is simply an allegory or fable. In fact, given that it is a fable about a set of data, we could call it a table fable. ®