This blog focuses on the science of designing surveys. Looking at the best ways to engage and communicate with survey respondents and how to generate effective feedback from your surveys. It explores and identifies new ideas and techniques surrounding online research. It looks at The Future of Market Research.

Thursday, 9 April 2015

Most business evolves in a classical evolutionary way. Through slow mutations in their approach to doing business, which leads to the business being more or less successful and in a survival of the fittest way - the strongest mutated variants win through. The most common way businesses “mutate” is through making a whole series of what is know as kaizen innovations – small baby step improvements and changes to increase the efficiency of a business.

Most kaizen improvement are logical evolutionary steps. If we do this we think we will make more money.

All of life evolves in this way, through trial and error. There are some interesting things thought that can happen if you break out of the evolutionary approach to development. Start to create things that never could emerge as a result of “market forces” of the demands of customers.

What do I mean by this, well the example I would like to use to illustrate this is the “unstable jet”.

Imagine a bird evolving a really really long beak, the longer the beak in theory, the more efficiently the bird could cut through the air and the faster if could fly. The problem with having a really long beak though is you reaches a point of instability, because a tiny fluctuation in the movement of the tip of the beak or a gust of air in a different direction and the beak could be deflected and instantly act like a sale and the bird would flip over its nose. As a result there are no birds with really really long beaks like this as they would be unstable.

However imagine a bird with a really long beak and a the end of it there was a sensor and small computerized navigation system that could make microadjustments to the direct of the “beak” to ensure that its always in a stable position facing directly into the headwind and not deflected off course by a gust of wind and now you have designed what in theory is a bird that can fly faster because it can cut through the air more efficiently. Unfortunately no bird is likely to evolove this extra step because the solution is “non-evolutionary”. It can never get there by baby step "kaizen" mutations. It takes a major new "non-evolutionary" improvement to get over the hurdle of an unstable beak.

Yet Man has been able to make this non evolutionary improvement that would have been impossible in the natural world and we have designed jets now with exactly this feature.

And this is the type of non-evolutionary form or innovation that I am particularly interested in.

Most of the establish businesses that are killed off are by major disruptive innovations like this – business solutions and leaps of improvement that break the classic evolutionary kaizen business development model. The step changes that existing businesses cannot make because it would result in the total cannibalization of their existing business (making them unstable).

To get there businesses need to take a completely different approach to innovation stop thinking about what would make money and start focusing on what is possible. What would happen if....To think more abstractly about what could happen if this other thing happened. To cross connect ideas. To build field of dreams. To invest in the connecting points. To look out for the non-evolutionary leaps.

Saturday, 3 January 2015

I can't change my survey as it will effect all our historical trend data.

This dilemma sits at the heart of most the discussions we have with companies wishing to update their research studies. "I would love to do things differently but I can't!"

And it is also the is the reason why so many of the surveys we look are behind the curve in way of design and questioning techniques,the reason why the average online survey length has crept up from 15 minutes to over 20 minutes over the last 5 years.

Well 2015 is the year in which things are going to have to change.

...and the reason is, our respondents are going mobile.

At the end of 2013 only 5% of people completing our surveys did so via mobile or tablet device, by the end of 2014 that figure has reached 20%. In some lead markets of Asia its already approaching 30% and as an indicator of where things are going, by the end of 2014 more than half the people signing up to our online panels did so via a mobile or tablet device.

What we are starting to see is stark differences in completion rates between those surveys that are mobile compatible and those that are not.

We are going mobile too...

By the end of this year all our survey respondents are going to have the choice of what surveys they want to complete and every survey that is not mobile compatible will be marked as non-compatible. As a result, the cost of fielding these non-mobile compatible surveys will start to increase significantly.

This is a change or die moment for many peoples tracking studies.

The days of getting away with a 20 minute+ grid dominated survey are pretty much over. The dropout rates of these types of longer surveys on respondents completing surveys mobile devices are over 50% , which is simply not acceptable for anyone.

Tuesday, 30 December 2014

We convert statistics into emotions: and so the best way to fast track getting your statistics remembered is to emotionalise them!

Our brains are Bayesian decision making engines: by and large designed to work out what choices will make us most happy

A question is a problem that you ask respondents to solve: it is easy to lose sight of this simple thought. Often we design questionnaires that skirt around the problem we are trying to solve. We ask questions so euphemistically, we ask questions that are a Chinese whisper away from what we are trying to find out.

We like to think in different ways: Researchers like to quantify things, particularly types of people and how they think and consumer. We have personalities that get classified and we are this type of person or that. The simple fact is that we are all sorts of different types of people depending on the time of day our mood and our circumstance. We all like to think in different ways, its boring the make the same types of decision especially when we go shopping. the concept of "type" in research is limiting. The same person who liked to try out new types of shampoo

Scale effects: you scale something up and sometimes different maths starts to apply

If there are infinite number of universes, then in some universes it is certain that a god will exist (as some of us know it) ..and in others it is certain that a god as we know it will not exist: A nice thought, that's assuming that there is such a thing as infinity, some physics question this too ... it might, but there is an infinitesimal small chance!

Rating something is inherently a system 2 thinking process: compared to a binary choice process which is more system 1. The example being you come out of a film your friend asked did you like it and in a fraction of a second you can say yes or no, but likewise if the friend asked you to rate the film this takes more mental processing and anything up to 5 seconds thinking to give it a score.

16 is a crowd: Prediction as opposed to market research is not about the numbers of people you ask its about the quality of information available to the group of predictors and their effort and objectivity.

Unwise crowds: crown wisdom is a nice idea but it only work in certain rather rare circumstances. Crowd predictions are mostly corrupted by system errors & network cognitive biases.

Computers using artificial intelligence can now read some of our emotions better than we can: Artificial intelligence tools are getting so good at reading our emotions by combining various input data sources ranging from how we are typing, the language we are using and the music we are listening to that they can identify traits of depression often months before it is apparent to ourselves

If you have any nominations do tweet them #mrjargon

Coming to the end of the year, I I thought I would share a list of the best books I have read in 2014 that I think other market researchers might like to read. Now not all of these are new books by any means so forgive me if you have yourself read half of them.

This will make you smarter

This book is a compendium of scientific and philosophical ideas in one of two page essays on quite a remarkable cross section of topics. There are some really exciting thought packed into this book that I think market researcher could make good use of. I think reading it really did make me a little smarter!

Expert Political Judgment: How Good Is It? How Can We Know?

Philip Tetlock's thinking has had more influence on how I think about conducting market research than any one person this year. I was introduced to this book by Walker Smith from the Futures Company and I would recommend that anyone who has an interest in the science of prediction should read this book. Learn that political experts are not quite as good as chimps tossing coins at predicting things!

The Signal and the Noise: The Art and Science of Prediction

I realise this book is a few years old now, and I wish I had read it sooner. There are so many really important ideas stuffed into this book that market researcher can use in their every day research. Its both inspiring and useful.

Strategy: A History

This small thumbnail belies a bloody thick book which I have to admit to not to have read every page of. It looks at strategy from all sorts of angles from war through to politics and summarizes the thinking of every major strategist in history including the likes of Sun Tzu, Napoleon and Machiavelli. There is loads of great thinking for market researchers to digest. And probably even more valuable incites for anyone running a business. It contains a detailed look game theory and the trials and issues with trying to apply strategy in real life. There is some sage advice in this book

Decoded: The Science Behind Why We Buy

This book is a really helps explain the basics of shopping decision making and is a compendium of behavioral economic theory, an important topic for nearly all market researchers to understand - I really like the way it uses visual examples to explain some of the theory making it an effortless read. This book should be on every market researchers shelf.

100 Things Every Designer Needs to Know about People

This book should really be titled, 100 things market researchers designing surveys and presentations should know about people! ...And everyone involved in either of these task encouraged to read this. Loads and loads of really clear, sensible advice.

The Curve: Turning Followers into Superfans

I read this after reading a very enthusastic linkedin review by Ray Poynter, thank you! It persuaded me to buy it. There are some nice radical ideas in here about how to market things by giving things away and at the same at the other end of the scale offering premium high price solutions for the those willing to pay for them.

The Numbers Game: Why Everything You Know About Football is Wrong

I rather immersed myself in reading sports stats books this year. The way that data is transforming sporting strategy, there are lessons to be learnt by the whole of the market research industry. As an English person with a love of football, I feel rather a bounden duty to promote the Numbers game which looks at how statistical data has changed how the game is played. I loved this book and I am afraid I bored senseless everyone I knew who had any interest in football quoting incites from it. I also read Money Ball this year too which is the classic opus on how a proper understanding of stats transformed the fortunes of a major league baseball team, it is a great story and lovely read.

Who owns the future?

This book has an important message about the impact of the digital economy on our future I cite from the book directly as it best explains "In the past, a revolution in production, such as the industrial revolution, generally increased the wealth and freedom of people. The digital revolution we are living through is different. Instead of leaving a greater number of us in excellent financial health, the effect of digital" Worth a read!

The golden rules of Acting

This is a lovely little book, you can read in one short sitting. Why though do I recommend market researchers read it? Well not because it teaches you anything about acting more about life and humanity and dealing with failure and the right approach to challenges. There is not much difference in my mind to going for an audition and going and doing a pitch presentation. I took some heart from reading this book.

Surveys are competing with a billion+ largely more fun things to do online these days and we are reaching tipping point were many people, the young age groups in particular are simply refusing to complete survey because they are too boring.

To have any chance of competing we have to change our approach and I think this starts with taking a long hard look at some of the boring questions we are asking in our surveys.

One of them being this one...

"What brands are you aware of?"

It a question asked in nearly every consumer survey I come across, usually asked both unprompted with an open ended question and then prompted with a closed question set of brand options (twice the work). Its one of those sacred cow set of questions that everyone insists on asking.

Do we actually need to keep asking this question? Are there not better questions that could be asked in better ways, that make better use of a respondents brain that deliver more useful data?

Here is the case for the prosecution:

Its a really dull question: From a respondents point of view these are probably the dullest most cliched question they continually have to answer in surveys. Respondents don't like answering these questions, they trigger drop out.

Respondents put little thought into their answers: Less than half the respondents you ask this question name more than one or two brand when asked unprompted and well over 20% say don't know. The prompted question often gathers together a random set of clicks from respondents.

Little statistical value: If the average respondent list 1 or 2 brand and assuming the number of individual brands listed adhere to a Zipf's law style distribution, the most popular brands named much more often than the least popular brands - on a typical sample of 400 respondents there will only likely to be 1 or 2 brands you ever have enough data to work with statistically.

Large error boundaries: The data error boundaries on this question on most survey samples are of often so large that in wave to wave in brand tracking studies the fluctuations down to pure statistical error are often of an order of magnitude larger than the actual underlying change in brand awareness. Resulting in a lot of "overfit"

No stand alone value: The data it delivers back is almost always duplicated or can effectively be modeled from answers given to other questions in the same survey.

Meaningless metric: Brand awareness information is totally useless in isolation from anything else. What it measures is intangible it's certainly not an accurate measure of purchase behavior for example - Unprompted awareness correlate at around about 0.54 with purchase behavior*, prompted awareness it drops to under 0.2*. If you specifically want to find out what brand consumers are likely to buy there are other question that are far more effective. It is also not a measure of how much I may like a brand....

I have never see how this information is really used profitably: It's always the chart that everyone skips past in a presentation.

The case for the defense:

Now I have challenged several prominent and respected researchers many of whom are still very wedded to asking this question in survey and asked them why they like to use it.

Its a fundamentally important measure: The brand that is mentioned first is the brand that has the most brain neuron connections and associations between the product and the category. So many see it as fundamentally the most important question you can ever ask in a survey.

Now I get this but I am still left with a feeling of so what....if you ask me what brand of chocolate I am aware of - I will say Cadbury's. I have been exposed to this brand all my life seen thousands of Cadbury's ads, seen the brand in every confectionery counter I have ever visited. But I never buy Cadbury's and so what x number of people are aware of a brand.

Now I am open to some other arguments as to why it should be kept if anyone wants to make them, but my judgement verdict is that its a question that if not completely banned, should at least be taxed, in the same way that cigarettes and alcohol are taxed to discourage their usage.

What could be asked instead?

There are a range of alternative ways of directly or indirectly measuring brand awareness that are more interesting and potentially useful.

I find one of the frustrations to answering the what brands do you recall question is thinking why does it matter and not knowing how many to list, Simply applying a rule to the question that contains the task it in a more meaningful framework for respondents can make it far less dull to answer.

you could ask:
...what are their favorite brands
...which brands they would have in their perfect supermarket
...which brand, if they could only buy one, would they choose to buy a life long supply of... ...which brands they would take to the desert island
...which brand they would invite to a party
...which brands they would recommend to their best friends
...which brand they would invest in
...which brands they think they will still be buying in 5 years time

All these will gather top of mind awareness to some degree or other but are more interesting and purposeful for the respondents to answer and adding in these "rule" can make them more salience and relevance e.g. asking about what brands they would have in their perfect supermarket it not just just a measures awareness but also intent to purchase.

In head to head experiments we have found we get more responses to these more conceptually fun questions too.

You could turn it into a full blown game by adding to your list of brands some fake ones and challenge respondents to pick out the real brands. We have used this approach on several occasions, its more fun for respondents and the data you get back is almost identical to prompted brand awareness. You could also show them a facet of the packaging of a brand and see if they can guess which brand its for, or show them a de-branded ad.

I would also advise jumping straight past the specific awareness question and asking what range of brands they have purchased on there last 10 shopping occasions. The has very high correlation with actual sales c0.8 and again cuts to the heart of the problem, if they are aware of it but not purchased it in any of their last 10 shopping occasions its not on their purchase radar!

If you want to defend the use of this question in your surveys and have got a strong argument for doing so i would love to hear your thoughts.

Sunday, 7 December 2014

As market researchers we like to classify people and in particular we
like to classify how people make decisions. However, we have a dreadful habit
of thinking that there are different types of people who think and make
decisions in these different ways. We define segments like loyalists and
switchers, impulse v considered shoppers.

The reality is that how we feel so often dictates how we choose; and our
ever changing mood states mean we are all manner of different types of shopper
all rolled into one.

All these different thinking process are all bundled together and
co-exist in a quantum style mix - we are all or nothing of these or a random
mix at any one point in time. It’s difficult to determine, and - just as
Schrodinger pointed out - just the process of observing changes things.
You ask someone why they have purchased something and it immediately puts
people into an analytical thinking framework, far removed from the mind state
they might have been in when they made the purchase.

The biggest problem for market researchers trying to understand decision
making processes is how to simulate these different moods to be able to
effectively measure these different choice situations and how to take account
of observational biasing effects.

Some mental states are easier to evoke than others. The price
conscious mood state and the impulse mood state are actually quite easy to
simulate in say an online piece of research. You just ask people to either shop
with a price conscious budget or ask them to perform the choices quickly.

Some of the others, though, are a lot harder to evoke. For example our
distracted shopping mind-set is an important one, as when we go shopping it’s
likely that we will often get distracted and have other things on our minds
while we do it e.g. I am having a relationship crisis and as I go round the
supermarket I am thinking mostly about that and making some choices on auto
pilot. What are these choices like? Are they the same as when we are, say,
in a hurry? I think not. The choices would probably be the more habitual ones
and could perhaps be more reward driven and may also be quite impulsive.

How you evoke these mind-set in a survey is a difficult one. If
you ask people why they bought something their rational thinking processes kick
in and what you get out is in effective cognitive dissonance - we can shape our
reasons for purchasing around what makes us feel good so often.

Don't get me wrong; this is useful information in itself. But it can
hide other less conscious factors that for many marketers are the factors we
are most interested in.

e.g. to say “I bought it because it was cheap” could hide some
resentment to the purchase. In a sense this statement is an excuse for buying
it or hide some guilty feelings about buying a brand that is viewed as
extravagant.

We are Bayesian decision making engines working out what would make us
happy

It has been observed that when we buy things we make a Bayesian trade
off prediction about what will make us most happy. On one side might be the
price, how long a product will last and the risks involved in making that
choice, on the other side are the benefits that the product delivers.

So to try and untangle some of this, I have been thinking about a new
research technique. A form of self-observed Choice ethnography,
where we get people to try to mentally map out all the different thinking
protocols involved in making a decision for different products. A sort of
helicopter viewpoint on their own behaviour, the sort of thinking that takes
place when you sit in front of a therapist and really try to self-observe your
own thoughts and feelings.

To test this idea out, I conducted a Gedankenexperiment, a
thought experiment on myself to see how easy it was to self-observe my decision
making protocols.

What I found was that the more I thought about it, the easier and more
interesting a process it became….

My Gedankenexperiment

I started to observe my own behaviour and traced out my decision making on
a variety of different types of purchases.

I started out by imagining my choice of beer in a pub, thinking about
it. Here are some of the potential factors that I decided were influential in
my choice:

What I fancy,

What choices they have available

What others are drinking

What I had last time

How much I had already drunk.

Interestingly, when I examined my decision making behaviour in a pub,
the first thing I learned, was that it was never about what it costs.

Even more interestingly I realized that when I go into my two local
pubs, in each one I am a completely different type of shopper: In one,
perhaps because it is more conventional with a limited choice of beer, I always
buy exactly the same brand of German lager. I never vary my choice of drink
ever...I thought about it and realized I am buying instant relaxation. That
brand has been ingrained into the experience.

In the other pub, which has a fuller range of craft beers, I never, ever
buy the same beer twice and certainly never ever buy lager. I take
pleasure out of trying different things and so I move from one pub being beer
monogamous to another where I am completely polygamous.

I then started to think about my choice of a beer in a supermarket. These
are the criteria that I process in my mind:

What they cost

What they have

What l like

What I have not tried before

What looks nice

In the supermarket the look of the beer became paramount compared to the
pub where it was the taste. Additionally, in the supermarket cost suddenly
becomes near the top of the list.

Then I thought about my choice of wine in supermarkets...

A weird thing about my wine shopping is that I realise I decide based on
trying to game the system as here there is actually too much choice... what I
do is I look at all the brands on discount and try to find the ugliest bottle.
My reasoning is that is probably the best tasting wine but it’s on discount
because nobody wanted to buy it because of the ugly packaging.

Next, I thought about my choice of shampoo. This boils down to:

What they have

What it costs

What I can bear to buy

I end up picking the cheapest brand that has the least offensive
packaging. But it often takes several minutes to decide.

Compared to choice of deodorant & toothpaste….do they have Dove/Colgate,
is it on offer, yes but my decision making time is seconds. In both these
categories I have made up my mind as to what brand to buy and have stopped
deciding.

This went on and I examined my thinking process across a wide range of
product purchases, trying as best as possible to observe some of the less
conscious factors.

I realise, for example, that my choice of confectionary was triggered
often by childhood associations. would
literally buy a chocolate bar to try and feel like I felt in a situation when I
was younger.

What have I learnt about thinking about my own behaviour?

1.How dramatically
varied it is, I could be described as both a loyalist and a switcher. From a
market research point of view it would be almost impossible to classify me as
one thing – and I am sure the same could be said for anyone.

2.I realise I was
able - by simply observing my own thinking process - to gather a range of
personal “incites”

I
buy old things in new places where I don't feel comfortable

In
new places where I feel comfortable I buy new things

In
familiar places I buy new things to break out of the routine sometimes

I
buy a limited portfolio of things if I am unsatisfied with the portfolio
on offer generally but hold resentments to the products I buy and so am
not loyal

I am
an extremely loyal purchaser once I have made up my mind about what is the
best product

When
I have not made up my mind I vacillate

When
there are more than one product I like in a category I sleep around, so to
speak, and can be very promiscuous (wine is a good example)

When
I am buying something from a category for the first time aesthetics
control so much of my decision making protocols - what it looks like is
key and I rely on design cue that basically say that product understands
me

When
I don't like any of the packaging this is when I become an instantly
disgruntled shopper

3.The process, once I got into it, was fun, easy and
quite cathartic, and I am sure with a bit of explanation anyone could use this
same process easily on themselves, but it does take time and thought.

I am left with the thought that this would be a
very interesting process to do on a larger scale, so that is what I am looking
at doing next….watch this space.

Friday, 19 September 2014

This blog post is a short introduction to the science of prediction which is a topic that I have been totallt immersed in over the last new months and recently presented about at the 2014 ESOMAR Congress with Hubertus Hofkirchner. I thought I would share some of what I have learnt.

The accuracy of any prediction is based roughly around this formula...

P Accuracy = Quality of information x Effort put into making the prediction x (1 - difficulty of accurately aggregating all the dependent variables) x The level of Objectivity with which you can do this x The pure randomness of the event

P = QxEx(1-D)xOxR

Here is the thinking behind this:

If you have none of the right information your prediction will be unreliable

If you don't put any effort into processing the information your prediction may be be unreliable

The more complex a task it is to weigh up and analyse the information need to make a prediction the less likely that the prediction will be correct

Unless you stand back from the prediction and look at things objectively then your prediction could be subject to biases which to lead to you making an inaccurate prediction

Ultimately prediction accuracy is capped by the randomness of the event. For example predicting the outcome of tossing a coin 1 time v 10,000 times have completely different levels of prediction reliability.

Realize that prediction accuracy is not directly linked to sample size

You might note as a market researcher, that this formula is not directly dependent on sample size i.e. one person with, access to the right information, who is prepared to put in enough effort, has the skills needed to process this data and is able to remain completely objective, can make as good a predictions as a global network of market research company interviewing millions of people on the same subject! I cite as an example of this Nate Silver's achievement of single handedly predicting all 52 US State 2012 election results.

Now obviously we are not all as smart as Nate Silver, we don't have access to as much information, few of us would be prepared to put in the same amount of effort and many of us many not be able to process up this information as objectively.

So it does help to have more than 1 person involved to ensure that the errors caused by one persons lack of info or another person lack of effort or objectivity can be accounted for.

So how many people do you need to make a prediction?

Now this is a good question, the answer obviously is that it depends.

It firstly depends on how much expertise the people making a prediction have on the subject individually and how much effort they are prepared to make. If they all know their stuff or are prepared to do some research and put some thought into it, then you need a lot less than you might think.

16 seems to be about the idea size of an active intelligent prediction group

In 2007, Jed Christiansen of the University of Buckingham took a look. He used a future event with very little general coverage and impact, rowing competitions, and asked participants to predict the winners. A daunting task, as there are no clever pundits airing their opinions in press, like in soccer. However Christiansen recruited his participant pool from the teams and their (smallish) fan base through a rowing community website, in other words, he found experts. He found that the magic number was as little as “16”. Markets with 16 traders or more were well-calibrated, below that number prices could not be driven far enough.

The Iowa Electronic Market, which is probably the most famous of prediction systems out there that has successfully been used to predicted over 600 elections, has I understand involved an average of less than 20 traders per prediction.

Taking account of ignorance

However for every one completely ignorant person you add into the mix who effectively makes a random prediction you will instantly start to corrupt the prediction. And in many situations this is scarcity of experts means to isolate ignorant and expert predictions this often means you need to interview a lot more people than 16.

Take for example trying to predict tomorrows weather. Imagine that 10% of the people you ask have seen the weather forecast and know it will not rain - these could be described as the experts and the rest simply guess 50% guessing it will rain and 50% not its easy to understand that if by chance more than 10% of the random sample predict it will rain, which is entirely possible the group prediction will be wrong. Run the maths and for 95% certainty you will need to have a margin of error of less than 10% to be confident which means you will have to ask 86 people.

It gets even harder if the experts themselves are somewhat split in their opinions. Say for example you were trying to predict who will win a tennis match and 10% of the sample are you ask are keen tennis fans (experts) who predict 2:1 that player A will win, the rest randomly guess 50% player A 50% player B. Because of division in the experts you now need to a margin of error of less that 7% to be 95% confident which means you will need to interview around 200 people.

Taking account of cognitive bias

It gets even harder if you start to take into account cognitive biases of the random sample. For example just by asking whether you think it will rain tomorrow more people will randomly say yes than no because of latent acquiescence bias. We have tested this out in experiments for example if you ask people to predict how many wine drinkers prefer red wine the prediction will be 54%, if you ask people to predict how many wine drinkers prefer white wine the number of people who select red wine drops to 46%. So its easy to see how this cognitive bias like this make predicting things difficult .

In the above example predicting the weather this effect would instantly cancel out the opinions of the experts and no matter how many people you interviewed you would never be able to get an accurate weather forecast prediction from the crowd unless you accounted for this bias.

This is just one of a number of biases that impact on the accuracy of our predictions, one of the worse being our emotions.

Asking a Manchester United football fan to predict the result of their teams match is nye on useless as it almost impossible for them to envisage losing a match due to their emotional attachment to the team.

This makes political predictions particularly difficult.

Prediction biases can be introduced simply as a result of how you ask the question

Imagine I were doing some research to get people to predict how often when a coin is tossed it is heads and I asked the question "If I toss this coin, predict if it will be heads or tails" for the reasons explained above the on average around 68% of people will say heads. The question has been asked in a stupid way so it delivers back a wildly inaccurate aggregated prediction. If you change the question to "If a coin were tossed 10,000 times, predict how often it would be heads" you probably need no more than a sample of 1 to get an accurate prediction. Now this might sound obvious, but this issue sits at the route of many inaccurate predictions in market research.

Imagine you asked 15 people to predict the "% chance" of it raining tomorrows and 5 of them happen to have seen the forecast and know there is a 20% chance of rain and the rest randomly guess numbers between 0% and 100%. If their average random guess is 50%, this will then push up the average prediction to 40% rain. If there is the same mix of informed in non informed predictors in your sample like this, it does not matter how many more people you interview the average prediction accuracy will never improve and will always be out by 20%.

This runs very much counter to how we tend to think about things in market research, where its nearly all about gathering large robust samples. In the world of prediction, its all about isolating experts and making calibrations to take account of biases.

The stupid way we ask question often in second hand ways we ask questions can exacerbate this.

"Do you like this ad" for example is not the same question as whether you think its going to be a successful ad. The question is a Chinese whisper away from what you want to know.

A successful ad is not an ad I like its an ad that lots of people will like. Change the question and motivate the participants to really think and we have found to make a perfect prediction about the success of an ad samples drop from around 150 to as low as 40.

Picking the right aggregation process

The basics

Imagine you were asking people to predict the trading price of a product and a sample of predictions from participants looks like this.

Now for this reason you cannot use mean averages. For basic prediction aggregation we recommend using Median. The median average of these is = $1 which looks a lot more sensible.

An alternative might be to simply discard the"outliers" and use all the data that look sensible. In this example its the $0.01 and the $10,000 that look out of sync with the rest removing these the medium average = £1.03 which seems a bit more precise

Weighting individual predictions

The importance of measuring prediction confidence

In the world of prediction its all about working out how to differentiate the good and bad predictors and one of the simplest techniques to do this is simply to ask people how confident they are in their prediction.

For example if I had watched the weather forecast I would be a lot more confident in predicting tomorrows weather that if I had not. So it would be sensible when asking people to predict tomorrows weather to ask them if they had seen the weather forecast and how confident they were. From this information you could easily isolate out the "signal" from the "noise"

The trick is with all prediction protocols to try and find a way of isolating the people that are better informed than others and better at objectively analyzing that information but in most cases its not as easy as asking if they have seen the weather forecast.

For more complex predictions like predicting the result of a sports match, prediction confidence and prediction accuracy is not a direct linear relationship but certainly confidence weighting can help but needs to be carefully calibrated. How you go about this it a topic for another blog post.

In the mean time if you are interested in finding out more about prediction science read our recently published ESOMAR paper titled Predicting the future

Thursday, 18 September 2014

Having spent the last few months researching and studying the science or prediction and also being quite fond of pub quizzes here is my guide to how to make a perfect guess in a pub quiz using some of what we have learnt.

Step 1:Ideation

Ask people to think of the first answer that comes into their heads.

If they think of an answer they should not shout out the answer, this could corrupt the purity of other participants thinking. They should put up their hand to indicate they have thought of and answer and write it down. The should also write down how confident they are on a scale of 1 to 3. Each player can think of more than one answer but they must score their confidence of each one.

Step 2: Initial idea evaluation

After the point at which everyone gives up, you then share your answers from the team and the level of confidence.

Rules for deciding if the answer is correct:

If more than one person has come up with the same answer in under 5 seconds then its almost certain that this answer is correct.

If anyone is certain about their answer, there is a high chance this answer is correct.

If more than one person comes up with the same answer and the combined confidence score is higher than 3 then there is quite a high chance that answer is correct and suggest you opt for that.

If there is a conflict or no answer scoring more than 2 point then go to step 3....

If nobody has come up with an answer the team is satisfied with go to step 4....

Step 3: Answer market trading

Each person must rate each answer by buying or selling shares in each answer choice with some "virtual money" . They can buy or sell up to 2 shares in each answer.

Tip: If a person has 2 ideas that both are "hunches" then the first idea research has shown this is around 30% more likely to be correct. Take this into consideration when making your buy / sell decisions.

e.g. if I think an answer is definitely correct I buy 2 shares. If I think its correct but I am unsure I buy 1 share, If I think its definitely not correct I sell 2 shares, If I am feeling a little uncomfortable that it is wrong I sell 1 share. Everyone has to commit to buy or sell - no body is allowed to sit on the fence.

Add up the total money traded in each idea and choose the winner.

If you want to be super nerdy about how you do this then don't simply add up the amount bet. Answer should be weighted somewhat as there is not a linear relationship between betting confidence and prediction accuracy. Having studied a data from a large number of predictions we have found that prediction accuracy of somone who claims to be very confident is not twice as good as someone who has a hunch its only about 20% better (see chart below). And people having a hunch are only 10% better than people making a total guess. Interestingly there is little difference between someone who has a hunch and someone who says they are fairly sure.

Further more when you look at people betting against things and comparing to betting for things the prediction accuracy of the amount bet varies in an odd way. Smaller negative bets are slightly more predictive we found than large negative bets. Strong positive bets on the other hand were more predictive than small positive bets but those that bet more than 2 were actually slightly less predictive than those that bet 2. Hence our 2 point betting scale.

A more accurate betting aggregation process should score the amount bet like this:

-2 = -20% -1 = -20%+1 = +10% +2 = +20%

If on either of these aggregation processes no idea has a positive trading value then go to step 4....

Step 4: Idea stimulation

If you are not satisfied with any answer, then all the team members should voice any "clues" they may be thinking about e.g. "I think his name begins with B" or "I think its something to do with football". Your thoughts could help another person think up the answer.

The scientific terms for this is called "Dialectical Boostrapping" - which basically means the sharing and discussion of ideas, which has been shown to help improve crowd wisdom generation processes. Find out more about this here Herzog and Hertwig (2009)

The more small clue you share they greater the chance of one of them triggering a thought in a team member. Note these can also be negative clues e.g. its definitely not...

If this process stimulates any ideas then go back to step 3 to evaluate them...

Step 5: Picking the best of a bad bunch of guesses

If you are left with more than one answer that nobody is particularly satisfied with, then pick the first answer the first person thought of. This one has the highest chance of being correct. It wont necessarily be right but it will have a slightly higher chance.

Advanced techniques:

Performance weighting your teams predictions

If you keep track of each individual's answer trading record over the period of several quizzes (.i.e if they bought 2 shares in an answer that eventually proved to be correct their personal balance would be +2). You can then start to weight your teams market predictions. You can do this by giving each person in the team a different total pot of money to bet based on their past performance record in correctly predicting the right answer based on how much money they would have won.

Note it would take several weeks studying at least 100 predictions to get a good idea of the prediction ability of each player so it would be a mistake to calibrate this after only one or two quizzes - luck has far more important role to play thank skill in the short term.

You might also want to assess how individuals confidence levels change when they have drunk 1 unit, 2 units 3 units of alcohol and start removing budget (or indeed giving extra budget!) as the night progresses!

Encouraging the team to think like foxes not hedgehogs

What buggers up the predictions of many pubs quiz teams can be the bullish viewpoint one or two individuals. Having a strong opinion about things generally I am afraid does not correlate very well with actually being good at making predictions. If you want to read up some evidence on this I recommend you order this book all will be explained.

The team should foster an atmosphere where its OK to change your mind, its not a battled between right and wrong , and should not be scared of failure.

Avoiding decision making biases

If the question is multi-choice make sure that your answer is no biased by the order effect or anchoring in the way the question is asked. For example yes/no questions more people pick yes than no for irrational reasons. When presented with multi choice options slightly more people pick the first choice for irrational reasons. By being aware of this you can be conscious that your decisions are being made objectively.

Important Note/disclaimer:

The advice is a fantasy methodology for making a perfect prediction. I don't advocate you using it in a real pub quiz. Firstly for practical reasons, In reality the speed at which most pub quizzes progress you probably would not have the time be able to implement this approach. Secondly it may also not be in the spirit of a fair pub quiz to use this technique in real life - it might be considered cheating!

Friday, 18 July 2014

Are you fretting about putting together a presentation for an up coming event? Here are some tips based on the experience of sitting through a few and delivering a few and designing a few. They are geared for market research but I suppose the thinking applies to any presentation.

Design

Images really really do help make a good presentation - but read presentation zen to understand how to use them effectively

Aim to present one thought per slide (you can break this rule if the slide is exceptionally well designed)

Avoid at all possible costs bullet points - this is as much a philosophy as anything - "set your baysian priors to zero", OK you might be persuaded to let one or two creep in but don't start with a presentation that is 100% bullet points!

Is your main message tweetable? Think about what people viewing your presentations will being doing - some may be tweeting the content so help them by turning your headlines into tweetable messages

Avoid video - don't fall into the trap of thinking adding a video will make your presentation more "dynamic" it usually dehumanizes your presentation.

Don't fret on look and feel too much - yes do your best but, there will be sure to be better designed presentations at the event you attend - the story and how you deliver the content is far more important - some of the best, most inspiring presentation I have ever listened to have looked awful from a design point of view (um...would it be undiplomatic to name Brainjuicer presentations as an example) - focus on the story and you will be fine!

Structure

Now this might sound a bit pompous advice, but to write a good persuasive presentation I really do suggest you first read up on the basic tenants of Greek rhetoric, in particular Aristotles ideas of Ethos, Logos, Pathos, the 3 ways to persuade.

You must start your presentation by establishing Ethos which is about building a bond of trust with your audience, then use Logos to which is about making a logic arguments and then end with pathos which is all about drawing out the emotions of the audience.

Often I see pathos being used wrongly at the start of a presentation e.g. kicking it off with some sort of cocky joke or dramatic video from which point everything else seems flat. Drama and emotion must be saved until you have won the trust of your audience and won the argument then you use it to drive home your message.

Ethos is really about establishing some humanity and connection with the people you are talking to on some level and have written about it in this separate blog poste

The logos part of the process is the most skilled and it is about identifying the key problems the audience might have on an issue and then outlining your solutions. Never present a problem to an audience if you are not going to follow with a solution.

Tips for writing the story

Your presentation must tell a simple story that you can recount in a basic elevator pitch

To devise your story really roughly sketch it out first either in your mind, on a piece of paper

Try writing your presentation as a story in excel - you will be amazed how effective this is at allowing you to coalescence you thoughts into a simple story - one line of the excel is one slide of your presentation. You can then once you got the basics down, really easily hack it around.

Go on a walk and tell the story to yourself or tell it to yourself as you are going to sleep or driving to work and see if it flows cleanly

If you get stuck telling the story to yourself you have what is known as a story knot - step back from it and try and tell it in a different way

Test market the story by trying to summaries what you are going to talk about to a colleague

Content market researchers should avoid

Don't pad your presentation with background stats - I don't need to know about all the details of the sampling techniques, we are all grown up market researchers - that's a given, just jump to the headlines.

Avoid the genero advice trap: we need to be faster, more insightful and cheaper!

Check if your content fails the mobile phone growth statistics bleeding obvious test: yes we know more people are now using mobile phones than own a toothbrush! Spouting any statistic we all could have a good stab at guessing or many in the audience has heard before is wasting delegates time.

If you are going to play to the crowd by highlighting one of the many short comings of our industries working practice, don't you dare propose a proprietary solution that you and only you can use. The audience will want to hit you!

Don't mention a problem without having a bloodly good solution to unveil that we can all grasp hold of.

Don't tell us you have a solution and then not show us the details or an example

If you are delivering a pure sales message about your great new piece of technology - come clean about it up front

I don't need to be told about how great Daniel Kahneman's work is anymore or need to be explained what system 1 and system 2 is.

Along the same likes, avoid the cliche buzzwords of the moment - in the naughties it was the word web2.0 which drove me mad hearing and this decades most important word to avoid using in any form of presentation is big data and I will let you make you mind up about all the rest

This is the year of the xxxx!!! mmmm... This is probably not the year of anything (and certainly not the year of the mobile!) Avoid protestation unless you are the chairman of the conference when that task becomes obligatory.

100% recycling someone else's ideas already recounted in a New York Times best selling business publication is cheating!

Technique

Try and make me laugh at least once (or at very least a smile)

Use simple examples: tell us your theory and then show us a real example could be seen as the essence of the structure of a good market research presentation

Be prepared to go off screen - Never under estimate the value of a good prop! Most of the show stealing presentations I have seen have used something other than just a presentation to get their message across.

Admit your short comings and failures in a sandwich between your successes. You could call this an integrity sandwich - we are much more open to hear about and believe in your achievements if you own up to your failures too.

Make it interactive - a quiz embedded into your presentation is the simplest way to do this but it can be tedious being challenge to guess the answer if there is no reward for doing so. Come armed with prizes if you are going to do this!

If you are going to get people to do things, make sure its inspiring. It embarrassing to all stand up so if you are going to ask your audience to do this it better be fun or genuinely interesting

Dress rehears your presentation the office first to your staff - they will benefit too by knowing what you are going to talk about

OK I suppose you better get your presentation spell checked too! As a class one dyslexic spelling is a challenge to me and often the first feedback I get when I give a presentation is a polite aside about the spelling. To avoid this type of humiliation I do recommend getting your presentation spell checked by a third party.*

*Admittedly this is not advice I always take myself to the fury of my marketing department and sales team as I would describe myself as a bit of a militant dyslexic and feel I have the human right to make spelling mistakes on my own content sometimes.

About the author

My name is Jon Puleston and I am VP Innovation of the GMI interactive division at Lightspeed Research. This however is my personal blog which means that the views I express here may or may not reflect the views of the company I work for! (disclaimer over). This is also a blog being written by a dyslexic, so I am afraid there may be one or two spelling mistakes to contend with so I apologise in advance.

The focus of my job is to work out how to design better surveys for our clients by making them more engaging consumer experiences. I have done a lot of research over the last few years into this topic and in the blog I hope I can share what I have learnt along the way. I am also interested in market research innovation more generally and so this is an area I also cover in this blog.