[Transcript of my talk from NewMR: Explode A Myth. The accompanying slide is here and you can listen to the recording of the talk and Q&A here]

If you’re the type of person who tunes into NewMR events, you can hardly have missed the library of behavioural economics books that have poured out of the typewriters of various academics in the last few years. Nudge, Thinking Fast and Slow, and Predictably Irrational have hammered into us the idea that we’re all irrational. We don’t make choices in our best interests and we’re always making mistakes. We don’t even know what we want. How can we be trusted to make our own decisions in life! We’re idiots!

Even the name of my own agency, the Irrational Agency, plays up this idea.

Today I’m going to tell you all this is nonsense. Consumers aren’t irrational at all. Not that they’re rational either…and thank goodness for that.

The fact is: if we really did behave the way economists say we should, we’d all be dead. The only people who make decisions in the way economists tell them to are people with specific types of brain damage. These people cannot use their intuition or feelings to make choices – they have to calculate all the costs and benefits of every decision they make. As a result, they can’t make any practical decisions in life: not even whether to get out of bed in the morning. They end up unable to live outside of an institution.

We’ve all probably experienced this on a small scale – they call it “analysis paralysis”. Any time you’ve found it really hard to make a decision between two options, and you’ve put it off and thought it over and worked it out and still been no closer to deciding – you’ve been in this mode. And eventually you probably picked one path or the other, and chances are it worked out OK.

Imagine you weren’t in a comfortable office (or living room) in a modern economy, but instead in a threatening environment, with only just enough food to survive on, and a wide variety of creatures waiting to kill you. In other words, imagine you were your great-great-great-great-grandmother 300 generations back. Now imagine you stopped to calculate and weigh up the potential costs, benefits and associated probabilities of every decision before making a choice. It wouldn’t be long before you ended up as lion food.

Any “rational” person – rational in the sense that economists would like us to be – would quickly be eaten out of the gene pool. Only those who are willing to make decisions quickly, find shortcuts and do what’s “good enough” would have survived. And those people are your ancestors, and mine, and the ancestors of the people who buy our products.

This isn’t just a matter of whether our brains are big enough, or if we just happen to be a bit too stupid and slow to calculate the right answers. It’s mathematically impossible for any imaginable creature to calculate all the pros and cons and predict the future consequences of their actions and weigh them up and make decisions, every minute or every second of the day. If you dedicated all the computers in the world, programmed by the smartest programmers, helped by all the people in the world and gave them a hundred years to calculate in exact terms the best sandwich for you to order today: they wouldn’t even be able to do that. There is so much information in the world; it’s such a complex place, and everything influences everything else; that nothing and nobody could process it all, even if given aeons to do it in. And certainly not in time to dodge that lion. It’s simply not an option.

So instead of calculating everything, we use heuristics. Heuristics, contrary to the impression given in some books, are not flawed or error-prone ways of thinking. They are brilliantly designed techniques to let you make a good-enough decision in less than a second, 99.9% of the time, without even being aware you’re doing it. The only reason we can navigate the world in a practical way is because we have these heuristics to make our decisions for us.

The heuristics are what let us choose products we know we’ll like. They’re what help us to drive to work safely, or talk to people without having to calculate and plan out every individual sound that every word will be made up of. It’s heuristics that tell us what brands we can trust and let us know we’re not being ripped off by a high price.

And that one time in a thousand that the heuristics don’t give us the right answer? Well that’s a better hit rate than any computer algorithm you can think of. Sure, algorithmic step-by-step calculations can give us the right answer to a small number of carefully defined mathematical problems: and there’s no harm in learning those situations so you know when not to use a heuristic. I wouldn’t recommend using a heuristic to calculate your tax bill. But don’t start believing the heuristics are wrong, or bad, or useless. And above all, don’t let them tell you consumers are irrational. Not even someone who runs an agency named after irrationality. (Sometimes I wish we’d just picked a nice acronym like GFK or TNS. But there you go.)

Don’t let your fantasy of that unattainable, perfect calculating machine be the enemy of recognising what a brilliant object your brain is, to have found its way past those lions on the savannah, through all the dangers of history, and past the supermarket till without buying the National Enquirer.

Consumers aren’t irrational, but you might be irrational if you think they are.

This week our partner Leigh Caldwell was interviewed by InDecision – a blog aimed mainly at academic researchers in the field of psychology of decision making. Every so often they feature a practitioner applying the science in the commercial world and previous interviewees have included e.g. Rory Sutherland from Ogilvy Change, Matthew Willcox from advertising agency Draftfcb and behavioural finance expert Daniel Egan from Betterment. (Original interview here.)

This week in our practitioner series we’re featuring Leigh Caldwell who is a behavioural economist and founding partner of pricing research consultancy Irrational Agency. He’s been applying decision making research commercially since the mid-2000s, making him quite an early adopter of this discipline, and is also active in academic economic research, working in the emerging field of cognitive economics. He has founded and run several businesses in technology and professional services, and recently condensed his experience in pricing and marketing these businesses into a new book The Psychology of Price. He is also the sub-editor for our upcoming interview series on applications of decision-making psychology in economics and public policy.

Tell me about your work: how does decision making psychology fit in it? I see my work as scaling up. I start from decision-making research that applies to individuals, and expand it into an understanding of how groups of people, companies or markets or whole economies, operate.

As a consultant, I do this for companies who want to know how to design a pricing or marketing strategy while taking into account consumer psychology. As a researcher, I do it with economic theory, building models of how markets work and how economies experience growth or recessions.

To do either of these jobs calls for mathematical models – models of how people behave which strike the balance between being psychologically realistic, but simple enough to work with. Old style economics went too far down the simplification route; but modern empirical psychology doesn’t produce simple models. So my work involves figuring out just how much simplification is enough, then doing the mathematics to expand it to an economic scale.

This field as a whole is called cognitive economics. Its goal is to build models of the economy that are based on a realistic foundation of how people really make decisions, and to bring an understanding of positive psychology into economic modelling – how people get utility or happiness from non-material goods, by modifying their cognitive state. It tackles questions like: what determines whether a company invents a new product or competes in an existing category? Why do companies make profits when economics says all profit should be competed away? Why are people unemployed? Why do people invest and save and borrow in the way they do? When people can get psychological benefit from intangible things, why do they still rely so much on material possessions? These are all really important questions which traditional economics can’t answer. Cognitive economics uses the discoveries of decision-making psychology to figure out why these things happen.

How did you first become interested in decision making psychology? I was running a business, a software company, and had been trying to work out for years how much money I should charge for our products. I could tell that my customers weren’t making decisions through rational cost-benefit analysis, so I wanted to know what else was going on. The same pattern showed up when we built software – whenever people started using it, they insisted on ignoring the “correct” processes and used it in whatever way they felt like. The Sheldon Cooper in me was frustrated by all this irrationality. I had to figure out what was going on!

I had read lots of marketing books with some foundations in folk psychology – anything from Dale Carnegie to Ries & Trout – but none of them seemed very scientific. As a mathematician and programmer with an economics background, my natural approach was to try to build a predictive model of people’s behaviour and figure out what was going on. When I started looking into the psychology research, I found out that there were plenty of researchers examining the same kinds of problems…but no coherent structure for how to apply the discoveries either to economics or business. That was where I discovered my niche. I decided to start applying this science, first in my software business and eventually set up a new business selling pricing advice. Having got involved along the way in academic research in order to find these answers, it seemed natural to keep working both on new research and on business applications.

What type of research do you find most interesting, useful or exciting? Like everyone, I’m entertained by the range of provocative topics people study in this discipline: the psychology of online dating, whether people called Michel are more likely to buy Michelin stock, or how easily people can be manipulated into saying the opposite of what they apparently believe – there’s always something fun.

But I’m always more interested in foundational work. This field is full of ad hoc papers, with lots of experiments focusing on individual standalone phenomena. Those are all fine in their own right, but they are hard to apply to real world problems. You need to do a new experiment every time you want to investigate anything. Theoretical work that unifies a spectrum of different results into a smaller set of principles makes it easier to solve new problems. That kind of work is what really fascinates me.

Do you see any challenges to the wider adoption of decision making psychology in your field? There’s resistance from the economics side of the discipline – many economists insist that people are fundamentally rational, even if they make occasional mistakes in their decisions. Their idea is that all the mistakes basically cancel out, or disappear once consumers learn to overcome them. That is part of why I want to turn all the disparate effects in this field into a unified theory: to find out whether our general cognitive limitations have an impact on the efficiency of markets or on whether societies end up rich or poor.

From the business side, the issue isn’t any direct resistance, just a lack of rigour and knowledge. Businesses are often run on superstition more than on evidence. The barrier here is inertia: a concerted effort will be needed to persuade companies and governments to take up these ideas. Fortunately, capitalism provides an incentive to make that effort – there are big rewards awaiting the agencies or consultancies who can win that role as a bridge from science to business.

How do you see the relationship between academic researchers and practitioners? Tenuous.

Two other interviewees in this series responded to this question with the word “symbiotic”. That’s true, but it’s also idealised. In reality, the culture – or to be technical, the habitus – of these two worlds are so different that it’s hard for them to work together. So far.

Academics mostly agree that it’s a good thing to make their work relevant for business or public policy applications, but many of them don’t have a clear idea of how to do that. (Business schools are a major exception – I’ve been impressed by the decision-making research conducted in the top business schools.) However, academics who are hired as consultants often struggle to make their work have an impact. Consultancy needs to be followed up by strong and simplified implementation steps in order to work, and academics rarely enjoy distilling their work in that way. Then again, that’s true of most commercial consultancy too.

Businesspeople are more skeptical of the potential for collaboration. No pharmaceutical company would deny the importance of rigorous biochemical science in creating their products, but it wouldn’t occur to most of them that decision-making science is relevant to their marketing and pricing too. I don’t think this means they’re anti-academic or anti-science, just that they don’t understand it and so it is easier for them to rely on gut feeling and intuitive judgment in this area. Quite a lot of my commercial work ends up being about translating scientific concepts into business language, and then demonstrating why business should be more open to using scientific methods and knowledge.

Mostly, the interface between the two worlds is limited to popular science books, a few intrepid people from marketing agencies who visit academic conferences, and the occasional consulting contract for a professor somewhere. I would love to be part of changing this. Right now we have two separate worlds and a few people who occasionally cross the border between them.

Imagine we could create a continuous spectrum instead: at one end theoretical academic research on mathematics or abstract models, then empirical research testing hypotheses, then an “engineering” discipline who knows the science and also how to implement it in business, through to marketing departments who use what the engineers have developed, all the way to operations or finance departments who could become aware of how to incorporate consumer psychology in the service they deliver and the way they bill for it. That would transform the practice of both business and academia.

How do we get there? Maybe we all need to apply some decision making psychology to understanding our own barriers and how to change our own behaviour.

What advice would you give to young researchers who might be interested in a career in your field? The areas I work in get their richness and value from the interplay of three disciplines: marketing, empirical psychology and economics. Researchers who are interested in pricing and other business applications will want to understand how people work in business as well as the scientific process of psychology and the modelling and mathematics that comes into economics.

For example, if you’re an economist and haven’t done empirical psychology work before, try getting involved in some pricing experiments at a business school so you can see how that works. If you’re a psychology researcher who hasn’t worked in a company, try working with a small business to try to redesign the pricing of their products or services. To get a feel for practical applications in pricing, you might also want to take a look at Priceless by William Poundstone (or my book). And if you’re a practitioner who wants to bring more science into your work, go along to some academic conferences or seminars just to get a feel for how people work.

Sometimes people are worried that they won’t understand the science (or the economics) or the maths will be too difficult. Try it anyway and just understand as much as you can. People in other fields aren’t any cleverer, they just speak a different language – the people who work in that field learned it, and you could learn it too if you wanted to.

Practical applications aside, if you’re interested in cognitive economics research, you will have to have an independent spirit. There are not many people working in the field yet, so you probably won’t find a supervisor who specialises in it. You can get in touch with me and I can help you identify a list of papers to start with, and then see what kind of research question you’re interested in. I think cognitive economics will be an increasingly important field and this is a good time to get into it; but it is always more challenging to work in an emerging field because the directions of research and the conventions aren’t clear yet.

Recently our co-founder Leigh Caldwell chaired the behavioural economics panel session at the Market Research Society conference. Our goal with this was to go beyond the anecdotal party tricks approach which has dominated much of the BE conversation in market research so far. Our title, “behavioural sillynomics”, was meant as a playful challenge to this conversation.

Of course party tricks are a fun way to introduce the ideas of BE for the first time, and they help to catch the attention of a new audience. The market research industry has already got to that stage, however, and is hungry to know how to apply these discoveries in practice.

We structured the session as a conversation between a client, behavioural scientists and – bridging the gap – market research agencies. The client wants to use BE to create more value for his consumers; the scientist has a whole library full of useful (but sometimes complicated) discoveries that she and her colleagues have made; and the agencies can sit in between, picking out the most important and practical pieces from the scientific literature and putting them into practice for businesses.

Our client, Erkan Balkan of PepsiCo Snacks, opened with a call for agencies to start finding out how consumers make decisions in an unconscious way. Existing research methods mostly rely on the conscious beliefs and memories of respondents, and rarely access the deeper drivers of behaviour. Some agencies have started to adopt BE methods in the early stages of research – qualitative methods which are usually used for ideation and invention. Even there, even the most advanced agencies only use behavioural methods for 20% of their insights. But to really change market research, we have to start applying BE at scale. The real money in research is in validation and testing, for ads, products, concepts or packaging – mostly quantitative disciplines. According to Erkan, nobody has really figured out how to scale up behavioural economics research. (Read more on Erkan’s views on Research-live.com.)

Barbara Fasolo of London Business School represented scientists and economists – the people discovering the underlying science. One possible reason BE has not immediately been seen as a useful discipline for market research is that it sometimes uses a straw man approach. BE sets up an image of people as “rational” in order to prove they’re “irrational” – and presents this as a revolutionary discovery. But no practising market researcher has ever been under the illusion that consumers are rational! Only economists believe that.

Instead, BE and the science of decision-making should forget the argument about “irrationality” and instead focus on providing a scientifically valid description of how people make decisions. Market researchers know a lot about decision-making, and behavioural economists know a lot too – by combining their expertise we can understand the consumer much better than before. The key insight that behavioural economists and psychologists can bring is an understanding of heuristics, and the smart methods that consumers use to find their way through a complex world. Barbara called this “Behavioural smart-onomics” which made a good contrast to our title.

So, a client wants to know how BE can help him, and a scientist has presented a new way to look at the discoveries the field has made. Could our two agencies act as a translation service, putting the science into terms that clients can use?

Lisa Edgar took us through an example of how her agency Big Window has done that. Working with client Jo Kenrick at Homebase, she measured the cognitive response of Homebase consumers to specific advertising messages. The psychology literature predicts that older consumers will rely more on “System 1” and emotional responses, rather than logical, considered “System 2” responses. This might suggest that emotional advertising works better on this audience (an important segment for Homebase). Lisa set out to test this, but it turned out that the picture wasn’t as simple as this. Older people turned out to respond better to logical, feature-oriented advertising than to emotional ads. But they took longer to do so, suggesting that they are alert to the risk of being fooled by heuristics and take care to think things through and avoid it.

Tom Vannozzi from Jigsaw then showed three case studies they’ve carried out. In one, they showed consumers messages about the social norms in their area – for example, that 60% of consumers eat the recommended five servings of fruit and vegetables each day. In a survey afterwards, they found that this message changed consumers’ claimed attitudes to healthy eating – although didn’t necessarily make them buy any more vegetables! In another study, subconscious “goal priming” of consumers changed their behaviour at a petrol station – if they were thinking of environmental goals, they’d make different choices than if they were thinking of goals around speed and excitement, or self-actualisation. Consumers weren’t conscious of how these messages or primes were changing their behaviour, but the effects were statistically robust.

The whole panel reinforced the message that there’s no real meaning in saying consumers are “rational” or “irrational”. Instead, we would be better off thinking about what a consumer’s goal is, knowing the heuristics they are likely to use to achieve it, how those vary by consumer type, and how to influence them. Heuristics follow certain rules – based on the capabilities and limits of our minds – and by knowing these rules, we can predict and understand consumer behaviour very well.

We also gave attendees at the session a copy of the handbook of behavioural economics written by our founders Leigh and Elina. If you’d like a copy, email us at info@theirrationalagency.com.

Smiley faces accepting the award from IJMR editor Peter MounceyEven smilier faces – it’s all team work even if there’s just one name on the prize!So shiny and pretty.

The paper itself will be published in the January issue of International Journal of Market Research, and all three finalists will be presenting their work at Young Research Writer Showcase presented by R-Net and IJMR on 28th January.

Behavioural economics keeps coming up in market research conversations. Surprisingly, I’ve never heard in that context one of the most important names in the discipline: George Loewenstein. Loewenstein’s work provides deep psychological insight into how consumers choose and value the products they buy. And one of his long-term research projects might have unlocked a key secret of why people want what they want: the theory of curiosity.

He is not one of the best known researchers among the general public: unlike Dan Ariely, Richard Thaler or Daniel Kahneman, he hasn’t written a book for popular consumption. But the scope of his writing and his research is as broad and rich as any of theirs. Ariely is well-known for clever experiments; Thaler for policy applications and experiments in behavioural finance; and Kahneman (apart from that Nobel prize) for coming up with some key simplifying models of thinking.

Loewenstein’s research has its own core message: people’s preferences are not straightforward. Despite the claims of standard economics, people seek out information or risk or meaning in a fundamentally different way to how they seek out apples or BMWs. This insight is probably more important for marketing than any of the better known behavioural economics discoveries. Since market research is traditionally focused on discovering people’s preferences (or as we call it in MR, liking), Loewenstein’s work is highly relevant.

Simplistic marketing, like simplistic economics, assumes that we can explain all consumer purchases in a simple way:

people have fixed desires

they know what those desires are and how strong they are

once they become aware of products that can satisfy their desires, they buy those products

In this model, marketing is just a matter of making people aware of your products in a memorable way; and market research is just a matter of finding out what their desires are.

Loewenstein’s research is all about the complex ways in which the things we want do not work like this simple model. He questions the relationship between our underlying long-term preferences and the product choices we make. He shows that it is not enough simply to understand what people like. To gain useful brand insight, we must also understand the process of how their subconscious minds construct choices from those likes, from their knowledge and from emotions, in order to finally pick a product.

He looks at how we trade off pleasure today against pain tomorrow, how our mood influences the choices we make, and how much we know about our own future behaviour. He looks at how our preferences and behaviour are controlled by ideas of fairness, how much we actually know about the attributes of the products we buy, and how much we enjoy the anticipation of consuming something, not just the actual experience of it.

His writing about all of this is engaging and readable, with plenty of fun examples. One of his experiments asks how much you would pay for a kiss from your favourite movie star – now, or in ten years time. Another paper examines why people risk their lives to climb Mount Everest while hating every minute of the physical experience of it. If you want the rigorous mathematical theory behind his ideas, that is there too – but you can skip it and just enjoy the writing.

For me, two particular subjects in his work are relevant to MR:

One of these should be required reading for all market researchers. His paper “A Bias in the Prediction of Tastes” tackles what is perhaps the most important question in MR: do consumers know what they want? Loewenstein demonstrates that they do not; and also that their mistakes are consistent (they underestimate how much they will value a product when they first possess it, but they don’t realise how much they will adapt to pleasure or pain over time). The paper poses a series of important questions that market researchers should ask about the accuracy of consumer responses, such as: Can regular buyers of a product understand the attitudes of people who don’t buy it? Do people realise how much their tastes will change in future? Two followup papers are particularly relevant to researchers in the food industry and other fmcg categories.

The other subject is one Loewenstein has been developing throughout his career. He has built a model of curiosity from a behavioural economics perspective. Why are people curious; why do we so desperately want to know things that will often make us unhappy when we know them; why do we so easily lose interest in something we were once intensely curious about, even without finding out the answer; and why do we care about information before knowing it (who killed JR? What does that person in front of me on the pavement look like?) that we barely give a toss about after we find out?

Curiosity in itself is an important subject. It improves the effectiveness of advertising, explains a lot about consumer behaviour, and has been explored by philosophers as one of the basic aspects of being human.

More fundamentally, Loewenstein’s answer to the curiosity question – that whenever we are conscious of an information gap, we instinctively want to fill it – provides the basis for a powerful general model of consumer behaviour.

Instead of seeing curiosity as a passion or a personality trait, he suggests that it’s merely one example of how we instinctively want to close perceived gaps in our environment. If this same drive applies to other gaps in perception, it could explain our desire for any product and the strategies we follow to satisfy that appetite.

This also implies a difference between wanting and liking something – with curiosity we may want to close that information gap even if we know we won’t like the outcome. What if the same applies to other products?

Richard Thaler may be best at understanding finance, and Dan Ariely at designing experiments, but I’d rate George Loewenstein as the top behavioural economist for illuminating the psychology of consumer behaviour in general. If you’d like to understand, and perhaps influence, consumer behaviour, it’s his papers you should be reading. Most of the papers I’ve referred to, and many more, are collected in his highly recommended book Exotic Preferences: Behavioural Economics and Human Motivation.

In the past year or so, the world of psychology has been rocked by scandals of scientific fraud involving fabricated data and dubious analysis techniques, among other things. The ensuing debates within the discipline have uncovered a host of other problems such as positive result publication bias, which means studies that have positive results supporting a hypothesis are favoured by academic journals.

Additionally, studies with surprising or counterintuitive findings tend to have a better chance of being published, which is good news for both the journal (as more people are likely to download the paper and earn the journal publisher some money) and the academic, who might get publicity in mainstream media as a result – not bad news at all for one’s CV!

Another recent and heated debate revolves around the proposed solution to the issue of scientific fraud: replication. The credibility of science as a whole rests on the replicability of its findings, i.e. whether another researcher is able to repeat a particular study and reproduce the same findings. If that isn’t the case, well, we should have less faith that the original research revealed anything meaningful about the world. The only problem is that academic journals are more interested in novel findings with a significant contribution to the discipline rather than publishing results that confirm (or disprove) the results of existing research while those with negative results which might disprove a theory are less likely to be published – leading to a bias towards studies which contain false positives.

Why is all this relevant to us as market researchers?

The rising interest in using behavioural economics and other findings from the ‘brain industry’ in market research has meant that more and more people are reading popular science books like Predictably Irrational, or perhaps the currently slightly less fashionable stuff by writers like Malcolm Gladwell or Jonah Lehrer. Lists of best social psychology books provide plenty of food for thought and are a great resource for getting to grips with these topics if you don’t have a background in psychology.

Popular books about social science tend to fall into two camps. Those written by journalists, who summarise research findings from a range of other people, and those written by the scientists who are actually carrying out the research. However, while books by “real” scientists like Ariely or Kahneman are just as likely to have been weaved into an engaging narrative, it’s less likely that the findings they report have been selectively cherry-picked and moulded into a story-study-lesson model to support their ‘big idea’ – a common issue with many titles in the section Waterstone’s now calls ‘Smart Thinking’. (For a detailed critique on the rise of “brain pseudoscience” in general, see here.)

Why does it matter?

We are naturally more drawn to simple narratives that make it easier to understand and remember things. This also makes us more susceptible to accept more engaging science stories as “truth”, simply because it’s easy to understand. However, most theories and concepts in psychology are not straight-forward or unambiguous, and usually involve numerous limitations on the generalisability of the findings.

While simplifying theories to make a topic more accessible to a wider readership is acceptable, there is a danger that, similarly to academic journals, only the most sensational and counterintuitive findings make the final cut, which can then distort or bias the conclusions we as readers take from the book. If we then let these ideas guide and inform our work as market researchers, we are introducing an additional bias into the work we do for clients.

So what should we do?

The most important thing is to stay critical of new research findings – especially the more sensational ones. Real science needs validation and replication before we can truly believe what it’s telling us about the world.

In the context of behavioural economics, there are even concerns within the behavioural economics academic sphere that some of the well-known effects and biases may not, in fact, operate in the way we have thought, or at least their existence may have been exaggerated by publication bias. But, if our only knowledge of the field comes from findings weaved into a narrative form, we remain unaware of the critique around these theories as well as what the limitations of the research might be.

Given all this, it’s a good idea to treat new theories and ideas with caution and question whether, given our own experiences of the world, they sound plausible or not. And if we really want to take something further and integrate it into the work we do for clients, it’s good practice to check how widely accepted a certain theory or idea really is and maybe even understand its limitations.

Yes, that takes time. But do we not owe it to our clients? Behavioural economics and other academic research has the potential to make our market research practice better, more accurate and more insightful – we just need to make sure we use it correctly.