Last month I went to the O'Reilly Strata Conference on big data and saw a similar pattern. Many customers were using big data tools to analyze Web user behavior -- to optimize Web app usability, shorten the path to online transactions, or serve the right Web ad at the right time. But quite often those customers were outsourcing both the analytics and the Web apps generating the clickstream data to be analyzed, rather than going through internal IT channels.

Whether or not IT leads the charge, the level of adoption is a fast-moving target. When I asked Smith what proportion of his customers have gone past the pilot phase and see big data analytics as something essential to what they do, I was surprised to hear him estimate 70 percent. Quite often in the past four or five months when he calls on a customer, "they introduce me to their big data architect."

No doubt this reflects the circles Smith travels in. A recent study commissioned by the U.K. firm Interxion seemed to indicate the opposite: Only 7 percent of businesses saw big data as a priority today. But a whopping 62 percent of the same 750 respondents believe it will become a priority within the next three years.

Smith told me that not just e-commerce operations, but brick-and-mortar retailers are embracing big data as a way of profiling customers and, ultimately, creating recommendation engines that will point in-store customers to special deals of interest to them in real time. Smith is clearly most excited by the possibilities in health care, citing a recent Seton Heart Institute pilot program where analytics were run on EMR data from patients who had suffered congestive heart failure to help predict who would be likely to be readmitted.

I've lived through several bubbles that have burst, but I don't think big data is one of them. Yes, there have been and will be clumsy attempts at big data analytics that yield little of value. But as expertise and software improve, the payoff will become more and more obvious -- not just for marketing, but for verticals such as health care, manufacturing, finance, media, you name it.

Some IT departments will embrace the experimental, iterative nature of big data analytics, and some will shrug it off as a fad. I'm betting that the latter will end up being less important to the organization they serve.