So does this justify the reported $125bn (or $50.1bn) (or $17bn) which is forecast to be spent on Big Data in 2015? Perhaps, but I doubt it.

I remember another time when data promised us a brave new world. In the olden days of the mid-nineties, the story, cited by sources such as the Financial Times, was how analysis of shopping baskets for a mid-range US store on Friday nights showed a previously unknown correlation between buying beer and buying diapers. This story and others like it made the idea of data mining credible as it promised a whole new world of customer insight and understanding.

The beer/diaper story, it turns out, wasn’t true, but it didn’t stop the stampede.

Companies spent fortunes on things called data warehouses, invested in stuff called knowledge management and we were all supposed to make lots of money by having greater insight into customer behaviour.

Consultants, vendors and the media who were all beating the data mining drum.

The consultants, who told us that this was the next big thing and that we would go out of business without it.

The vendors, who sold us the kit and the software and the maintenance contracts and the patches and who told us that the issue we were complaining about “…was a known bug that would be fixed in the next release…”.

The journalists and the advertisers, who sold ad space and wrote articles about “Making the case for knowledge management” and held conferences with titles like “Data Warehousing: A strategic imperative” or some such.

So we went to the conferences and read the articles and called in the consultants and bought the kit and installed the software and did the management of change and ran the benefits realisation exercises and employed new people with strange job titles and tried, as the dust settled, to see the step change in performance promised to us.

While the drum-majors walked away to do the same thing again with new clients.

Result? Billions of dollars siphoned out from the pockets of companies who were doing real things for customers and into the coffers of these drum-beating companies.

At the time, I recall that the consultants, tech companies and media companies in this space all posting phenomenal revenues.

What I don’t recall is reading so much about the great profits their clients made from all this knowledge management and data mining.

As far as I can tell, the promise of data mining and knowledge management was a promise unkept.

And now we have Big Data.

Perhaps Big Deja Vu might be a better term, because it all sounds very familiar.

The scale, depth, density and timeliness of customer data available to us is magnitudes greater than ever before. Mobile data, geolocation, behavioural antecedents, digital payments, social media and evolution of the web are streets ahead of the consumer purchasing information upon which the promise of data warehousing used to rely.

I just feel we need to think harder about how we want to take advantage of it. The same people are beating the drum and more and more companies are falling into step.

But the drum beat isn’t our drum beat. It’s the beat of market hype, of technologists seeking a market, of consultants seeking The Next Big Thing.

It’s not the drum beat of the customer.

What customer insight are we missing? Why does it matter? What could we do better for customers if we had it?

And (this is the kicker) what difference will it make? Really?

If we are to avoid being vendor victims, we should begin – as always – with the customer. And if we can answer these questions properly, then perhaps Big Data may really be of value to customers and to the companies that sell to them – and not just to those who are selling tickets to the Big Data bandwagon.

It is only a matter of time before some sites simply use our social identity, online behaviour, some statements of preference and history of other purchases to predict the best purchase solutions for us and offer a focused choice that’s right for us, based on who we are.

More than a search based on budget, distance from home, type of hotel preference and preferred flight times, this is a search based on what we genuinely want and like, evidenced by our behaviour.

This is, I believe, the kind of thing that will be enabled through the data gathering and machine learning capabilities of facilities like Google Now or Amazon Echo. I think we will start to see these services being offered in 2015.

Before long, we won’t be typing in “…washing machine reviews…” when we want to think about replacing our white goods. We’ll simply muse out loud, in our living room: “Alexa? What washing machine would be best for me?”

We will start to see fresh food delivery online at scale from non-grocers

Building a distributed, refrigerated real-time supply chain to distribute fresh food is expensive and difficult. This has acted as a major barrier to entry for online players. It has kept traditional grocery supermarkets in the game and let them sell us all the other household stuff we need routinely.

At least one traditional supermarket will experiment with a alternative models

Of course, traditional supermarkets are no mugs. They will experiment with new ideas to ‘lock in’ our weekly shop and keep it away from online pretenders. Tesco already have a subscription model for online delivery charges. I would be surprised if they, or their competitors, didn’t come up with an alternative. A single monthly subscription that delivers a weekly food shop, for example?

Health insurers will offer discounts for customers to upload their health data activities.

This one is, I think, already happening. Wearables, and the health monitoring facilities offered by the iPhone 6, all gather data about our long-term lifestyles.

Health insurers are beginning to incentivise us to upload this data to the cloud. Health insurers will analyse this data and give us bespoke cover at tailored premiums. And we will like it. (unless we’re fat, or sedentary, in which case our premiums will shoot up).

Wearable tech will cause at least one Big Data / privacy scandal

Wearables are probably the first tech innovation designed from the ground up to enable ‘Big Data. Just by wearing a watch or wristband, our movements yield long-term telemetry data.

Where we go, what we do, who we meet, what we are interested in, when we do things, how we get there, what we buy, how we pay and who we tell – this so-called ‘digital exhaust’ trails behind us as we live.

Buyers of customer experience management software will get wise to the idea that there is no such thing

Leading, managing and operating an organisation to offer a good customer experience is not about software. It is a matter of principle, strategy, practice and attention. I expect that more companies will understand this in 2015.

Banks will claim to be customer-centric but scandals will continue to disprove this idea

But I fear that this hope will founder, for two reasons. First, a typical bank IT system is a messy legacy stack that is hardwired around products, not customers. These systems are too big and too complex to change without eye-watering expense.

Second, the majority of banks continue to reward their people to focus on short-term revenues for the bank, regardless of the interests of the customer. Because these rewards are so big, they will continue to distort banker behaviour away from the customer.

Some brands will begin to function as marketing algorithms

The lines between ‘pure’ marketing and customer experience continue to blur. Marketers have been trying for years to personalise their messaging to individual consumers. I think this year, we will start seeing the first marketing content, bespoke for individual customers, automatically generated by what marketing systems know about each customer.

What will make these communications different? They will be driven by specific parameters that properly reflect the intent of the brand.

While the content of all these emails may be different for EVERY customer, each will reflect the presentation, tone of voice and content of the brand. The brand will have become an algorithm, driving content.

What are your predictions?

So these are my predictions for the year. Things will, I think, get better in incremental ways for the customer as more companies recognise the competitive advantage this gives them. We’ll see a few new things, and some ideas will fail (and that’s ok, failure is the overhead of innovation and the cost of progress) and a number of things will surprise us.

What do you think? What do you predict customers will see differently in 2015?

*The link takes you to a cool interactive tool by Google that shows how the purchase journey varies for customers in different market segments and countries. Fun to play with (if you like that kind of thing).

Attitudinal marketing is dead Well, if not dead, then it’s about to enter life-support. The quantity and predictive value of behavioural and activity data means that what people think or feel about a product or brand will become increasingly irrelevant. We are already finding this on the web. If A/B testing shows us that consumers prefer to press a red button, and not a blue one, then we are better served by changing all our buttons to red than spending a fortune trying to understand why. This thinking will soon apply everywhere.

Prepare for the segment of one Big Data will enable us to direct contextual, customised marketing directly at individuals based on such things as (say) their mobile GPS history, online and social media activity, and offline behaviour. In effect, a marketing campaign for one person. One implication of the segment of one is that a consumer marketing operation may well need to deliver a million tailored campaigns a year.

This is not just an automation problem.

To run at this level, with minimal errors, cost-efficiently, means the winning marketing operations will be those which adopt and implement the Lean manufacturing disciplines which enabled car manufacturers to deliver a batch size of one, with a cycle time approaching zero. (See my earlier post – SMED: The secret sauce of customer experience, for a related discussion).

We are all going to become Lean, people.

Create platforms, not campaigns The role of the creative will change. Increasingly, we will need our creatives to design communications platforms, rather than individual campaigns. These platforms will have to flex in innumerable ways to meet the contextual demands of the segment of one.

Brand as algorithm Brands will be formulated into heuristics – rules which can drive real-time decisions to enable real-time marketing. The automated brand is coming.

Source, don’t build, your data By definition, Big Data is a mix of different data sources. Very few organisations have the capability to assemble, structure and support such heterogeneous sets of data and stay sane (and profitable). Ignore the Big Data hype about the need to build Hadoop clusters and recruiting data scientists. This isn’t how it is going to go.

Here is how it might. Companies are going to realise soon that they will be better off working with trusted data intermediaries rather than trying to build their own Big Data. They will pose questions to these intermediaries, such as “….what is the best way to segment the market to identify the people most likely to buy our stuff?…”, or “…when in the customer’s day are we most likely to get positive attention for our proposition…?” or “…who could be our next customers….?”

These intermediaries will orchestrate data sources quickly to get the best answers to these questions. They may already own some data, some data they may rent, some they may commission and some will come from their clients – but such tasks are best left to specialists. There is no need to build your own data engine. Spend your time instead trying to understand the questions you need to answer to get to market most effectively.

Of course, for companies which specialise in data harvesting, brokerage, mashup and orchestration, this intermediary role will be a lucrative opportunity. For the rest of us, being able to use such services intelligently will become an increasingly important skill.

Big Data is going to change marketing. But those marketers who do embrace this change will become hugely more effective, productive and influential.

Malte Spitz is a member of the Bundestag, the German parliament. He sued mobile operator T-Mobile to get their records of his cell phone activity for a six month period in 2009. It came in an Excel spreadsheet with 35,851 rows.

Zeit Online, the digital imprint of Germany’s top-selling weekly newspaper, Die Zeit, combined this data with other information about Hr. Spitz’s life which they gleaned from social media and publicly available online sources.

“Each of the 35.831 rows of the spreadsheet represents an instance when Spitz’s mobile phone transferred information over a half-year period. Seen individually, the pieces of data are mostly inconsequential and harmless. But taken together, they provide what investigators call a profile – a clear picture of a person’s habits and preferences, and indeed, of his or her life.

This profile reveals when Spitz walked down the street, when he took a train, when he was in an airplane. It shows where he was in the cities he visited. It shows when he worked and when he slept, when he could be reached by phone and when was unavailable. It shows when he preferred to talk on his phone and when he preferred to send a text message. It shows which beer gardens he liked to visit in his free time. All in all, it reveals an entire life.”

I will leave it to other commentators to discuss the political, legal and ethical issues raised by Big Data. I am going to assume, instead, that it is here to stay and that it will increasingly affect our lives.

In my next post, I will develop further some ideas about how Big Data will affect Marketing.

If Big Data delivers what it promises, then the implications for Marketing – and indeed, all of business – will be profound. Before we can understand what these implications might be, we first need to understand what Big Data actually is.

Big Data makes big promises. But many of these promises have been made before, with, for example, data warehousing (and the famous beer and diapers story). What is different this time, and what difference will it make?

I believe, however, that most of us with a business perspective need a definition which enables us to think about the value and uses to which Big Data might be put. My stab at doing so is below, borrowing freely from excellent recent articles by Hung Lee (Big Data is Not ‘Lots of Data’) and Alex Cocotas of Business Insider.

Big Data is not (just) big data

What makes Big Data interesting is not the the size of the data sets (although these can be mind-bogglinglybig). The value of Big Data is much more about the kinds of data which it embodies, and (particularly) the uses to which it can be put.

Big Data is unstructured data.

Big Data is not that which fits neatly into a relationship database or which can be categorised by tags (although it may well contain data of this type). Big Data is a hybrid mashup of different kinds of data such as geolocation, contextual data, telemetry, life events, video, demography, social media and more. It’s messy, complex and has fuzzy boundaries.

Big Data is behavioural, not attitudinal.

It is about what people do, not what they think. Big Data is not, for example, about focus group findings or survey results.

Big Data is about small interactions.

Big Data might include transaction information, such as what is in our shopping trolleys – but this is a known game and is only an adjunct to the important stuff. Important stuff? What we do before we put things into our trolleys. Where we have walked. Whom we have met. What we do for fun. What the weather is like. What we are wearing. What everyone else is doing. The TV channels we watch. You know: the small stuff we do all the time.

Big Data changes. All the time.

Big Data is gathered continuously, in real-time, often from millions of dynamic sources. This means that at any time, we can only have a snap-shot of this continuing river of data. By the time we look at it, it’s already changed.

Big Data is online, mobile and the real world.

Big Data is credible because Google and Amazon and (a very few) others have been able to farm and use complex online customer behaviour data to make serious money. Now mobile is changing the game. Those of us with smart phones use them everywhere. We use apps to help us in the physical world. We use services like GPS and geolocation which note everywhere we go. Now when we turn on our mobile phones, we create data about our behaviour in the physical world in ways comparable with the data we create online. What do we call this melange of online, mobile and physical information? Big Data.

Big Data is informational debris.

Big Data is what we throw off when we do other things. When we stop what we are doing to fill in a form, or have our picture taken or scan some stuff to get ourselves registered or updated – that is not Big Data. Or if it is, it is only a small part of it. Big Data is what we leave behind us when we play games, or take pictures, or move house, or phone someone up, or browse around a shop, or go for a run or change the channel. It is a side effect.

That’s my shot at defining it. Does this work for you? What have I missed or got wrong? Let me know.

If Big Data keeps its promises, the implications for all of business – but especially Marketing – are profound. I will explore some of these implications in my next post.

This series of posts arose as a result of a panel discussion earlier this week at IQPC’s CMO Exchange event at St Albans. I had the pleasure of sharing the platform with Paul Blacker of BT and Michael Woodburn of Capital One, and it was admirably chaired by my old chum Vincent Rousselet, CEO of the Strategic Planning Society. Our conversation offered a good range of views on these questions. These opinions I express here, however, are entirely my own.