1/27/2013

I have been discussing what I learned in a podcast in which Jim Manzi cautions people to be very skeptical about the authority of conclusions reached by social scientists, including economists. The point he consistently makes is that, to have reliability, any phenomenon should be repeatable in different situations in different contexts, allowing people to make consistent and reliable non-obvious predictions. One important corollary is not to make too much of a single study — no matter how interesting the result might be, or how much it seems to confirm your pre-existing biases.

We all do this. Here’s a fun real-world example showing why it’s a bad idea.

Economic theory tends to hold that greater choice leads to greater demand and consumption — thus, if you want to sell more, offer more choices to your consumers. But Manzi tells the story of researchers who set out to test this. They ran an experiment in which they set up a table at a grocery store on two successive Saturdays. At the table, they had a selection of jams and jellies. One Saturday they had a selection of six different jams and jellies, and the other Saturday they had 24 varieties. On each day, they asked shoppers to taste their wares, and if they liked them, the shopper would be given a dollar coupon to redeem at the checkout counter to purchase one of the jams or jellies.

Conventional economic theory would hold that the day where the greater choice was available, a higher percentage of coupons would be redeemed. But the researchers found something counterintuitive and interesting. On the day where they had six different jams or jellies for purchase, fully 30% of the shoppers used a coupon to buy jams or jellies. On the day when they had 24 different varieties of jams or jellies, only 3% of shoppers redeemed the coupon.

Fascinating, huh? Reducing choice actually increased sales. You can easily see journalists writing an article that wows the public. Call the Freakonomics guys. Can’t you envision a section of a chapter talking about this surprising result, and discussing the likely reasons for it? Perhaps the shoppers were paralyzed by indecision when presented with so many choices. Valuable information, certainly, for any marketer to know.

Valuable — and almost certainly wrong.

Manzi says it was a mistake to draw sweeping conclusions from this single experiment. After all, look at how extreme the results are.

Can it really be true that all you have to do to increase sales by a factor of 10 is to remove 75% of your supply from your shelves? If this is the case, Manzi says, retailers everywhere are leaving “suitcases of money on the ground.” That is a HUGE effect — and frankly, so remarkable that it should raise a red flag.

The effect was so remarkable, in fact, that people have tried to replicate this experiment using other products, in other contexts, to see if the results can be repeated. And they can’t be. Manzi says that the experiments produuce a wide range of distributions — and that the general trend is that greater choice leads to greater demand.

Remember this the next time a single social science study comes to a conclusion you like. It’s tempting to cite such a study — we’re almost all guilty of this. But I am personally going to try to be aware of this in the future. Unless a phenomenon is proven through repeated observation in different contexts, allowing people to make repeatable, reliable, non-obvious predictions, the result of a social science study should always be viewed with immense skepticism.

This is an absurd straw man. The silly argument that Manzi wants props for demolishing was touted in a book by a sociologist, Barry Schwartz (The Paradox of Choice). I don’t think that more than a tiny number of economists looking for heterodox things to write put any stock in it.

There are tons of “studies” out there. Some are very carefully done, others are obvious crap. A dozen poorly done studies aren’t as persuasive to me as a single well-executed one with credible data.

A pretty good rule of thumb is that if the “study” is being put out by a “policy institute” of any political stripe, you should be wary.

Patterico–any chance that you might consider repositioning the part 1, 2 and 3 to the beginning of the thread name instead at the middle or end? It’s getting kind of confusing and cluttered over on the “recent comments” side of the screen.

Most social science studies begin with a premise which is soon validated by those selfsame studies. Call it the Dirty Old Man Rule: social scientists rarely falsify studies in which they have an ideological, emotional, or financial interest.

This is an absurd straw man. The silly argument that Manzi wants props for demolishing was touted in a book by a sociologist, Barry Schwartz (The Paradox of Choice). I don’t think that more than a tiny number of economists looking for heterodox things to write put any stock in it.

He doesn’t want props for demolishing it. He is using it as an example of how ONE study in a social science discipline cannot possibly be considered reliable because there are so many potential variables that can make the results questionable.

It’s a good point that you seem to overlook when you say you’d be willing to put stock in a single study if it had “reliable data.”

What you should get out of this, is that observational studies can’t be trusted. This is true with food studies where clinical trials often debunk observational studies. The jelly study falls under observational study. Here’s a good video on what we should be looking out for on food observational studies. I believe you can apply it to other observational studies.

I haven’t made it in a long time but I used to love it. Cut the cream cheese block in half (either direction, depending on the size of the plate you’re going to serve it on), put the wasabi paste in the middle, and then put the cream cheese blocks back together. Roll in toasted sesame seeds, and pour soy sauce over the top, garnishing with chopped green onion, cilantro, and/or minced fresh jalapeno. I like to use Triscuits but you can also serve it with rice crackers.

“Conventional economic theory would hold that the day where the greater choice was available, a higher percentage of coupons would be redeemed. But the researchers found something counterintuitive and interesting. On the day where they had six different jams or jellies for purchase, fully 30% of the shoppers used a coupon to buy jams or jellies. On the day when they had 24 different varieties of jams or jellies, only 3% of shoppers redeemed the coupon.”

This is well known in medical decision theory. The New England Journal published a study about 20 years ago supporting this. If you offer a patient with arthritis a choice of several drugs for arthritis, they are less likely to fill the prescription than if only one or two are offered.

There is a lot of decision theory research on this topic. If a student is on his way to the library to study and is met by another student who invites him to a party, the first student is less likely to do either and more likely to just go back to his dorm room.

It’s why car salesmen show the customer only one car at a time. Lots of research on this.

The case that free markets lead to the most efficient allocation of resources is completely dependent on the idea that consumers can correctly judge what is in their own self-interest. Because the welfare that is maximized in a free market is determined by that distribution of resources that maximize consumer consumption over all consumers. Arguments that question an individual’s capacity to make consumption decisions are nothing more than a backdoor into tyranny. The free market presumption about the individual is simply that he will pursue his own happiness as he sees fit. That fact that others might question his choice should be of no consequence. But this flies in the face of our therapeutic liberalism. Of course the bright people who went to Harvard can do a better job of choosing for the masses. That’s the whole point!

Sadly, a lot of this is based on a consensus that the social sciences have much to tell about how we should govern, or more aptly be governed. Most social science “experiments” are not worth the time spent by the subjects let alone the researchers who devote entire careers to these charades IMHO. About the best thing that can be said of them is that they inspired the opening scene in Ghost Busters, and for that they will forever be of redeeming social value.

However, there is a new kind of experiment that utilizes advanced technology, extensions of the now-familiar MRI are one example, that can begin to sort out how we actually process information. Leonard Mlodinow’s recent book, Subliminal: How your unconscious mind rules your behavior surveys these types of experiments and the results are fascinating. The first few chapters deal with the new technology and one of the startling results is that certain kinds of information will trigger the subconscious to change the path of information processing within the brain. His example is wine tasting using fMRI (Functional MRI) and the discovery is that telling a subject that one wine is very expensive, while the other is of only modest price, will change the parts of the brain that are active while tasting the wine samples, with the result that the expensive wine always tastes better … even when both are from the same bottle. Mlodinow also has a number of results relating to memory that are probably already familiar to all the lawyers and prosecutors who seem to populate this blog. But I found these results to be more subjective, less convincing, and tending to the old school of social science. Indeed Mlodinow strays far from his basic theme when makes the argument that global warming skeptics are suffering from some form of irrationality since there is a consensus that AGW is true.

The question of the week is, of course, What difference does it make now? Before the federal government inserted itself into everyone’s life, it was quite respectable to have an eccentric uncle who squandered his inheritance on expensive wine that he thought exquisite, and ended up living with his older sister as the livein gardener into their dotage. But this is no longer tolerable. Foolish people are costing the government $billions and something must be done about it! I am always amused by doctors who complain about the irrational choices their patients make, and the apparent need for wiser people to make their choices for them. Presumably the clever overseers called for in this view of humanity will do what’s best for their clients, but history is not reassuring. Eugenics comes to mind. At any rate, my feeling is that even if we all make irrational choices, it is far better for us to have the freedom to make these choices than not. Mainly because limiting everyone’s choices to those setforth on some menu prepared by the federal government will kill off innovation, and with that, all hope for the future. I will even support Mlodinow’s right to irrationally conform to some alleged consensus even though he is amply trained to examine the scientific arguments and data, and then to judge for himself. He likes to get along, and that’s how he’s built. He will make his contributions in his own way.

The Declaration of Independence said that the revolution was necessary to allow Americans to pursue their happiness. This was not a idle choice of words. Happiness is perceived by the individual, and it is different for us all. When each of us has the liberty to pursue our own happiness, then our country will prosper.

Did they sell less jelly or just redeem less coupons?
Did they sell more peanut butter than usual?

Just kidding around. But I think it comes down to common sense. We all want a variety of choices… but we want those choices to come to us at a pace we can process easily.

I’m just about smart enough and have the palate for five choices of jams and jelly.
strawberry, peach, razzberry, blackberry, and blueberry. Tell me they all come seedless too and tell me all varieties are also offered in a sugarfree product. Tell me they’ve got peach, apricot, orange marmalade, etc and let me try them if I ask, But don’t put them all out on the same frickin table at once because I don’t have the time or palate cleansing ability to try them all. I’ll get exasperated within the process and just give up on you until you learn how to market multiple choices in a way that doesn’t overwhelm my palate and/or my patience

“One Saturday they had a selection of six different jams and jellies, and the other Saturday they had 24 varieties. On each day, they asked shoppers to taste their wares, and if they liked them, the shopper would be given a dollar coupon to redeem at the checkout counter to purchase one of the jams or jellies”

There is a time cost in testing and being rewarded for my efforts. I am more likely to spend time on a short survey for reward than sticking around and debating the merits of 24 different varieties. I have a family time schedule to keep, or places to be.. or things I’d rather be spending my time on. How much do you value your time in exchange for a $1 reward?

Economic theory tends to hold that greater choice leads to greater demand and consumption — thus, if you want to sell more, offer more choices to your consumers.

Really? And what “economic theory” holds that this little jewel is true? (Or is “economic theory” being used instead of the concept, assertion?) Since Mr. Manzi’s experiment demonstrates that this obviously isn’t the case, one wonders how valid is the “economic theory” that is implied. I mean, no one ever bothered to test such an assertion before creating an “economic theory” around it? Sheesh…

It’s garbage in, garbage out. Studies — even economic ones — can definitely prove certain things. However, the proof is all dependent upon how well the argument is constructed and to the degree from which the conclusion necessarily flows from the premises. Poorly conducted studies with broad, shaky conclusions are worse than worthless.

Wherever she goes, people tell Iyengar about her own experiment. The head of Fidelity Research explained it to her, as did a McKinsey & Company executive and a random woman sitting next to her on a plane. A colleague told her he had heard Rush Limbaugh denounce it on the radio. That rant was probably a reaction to Barry Schwartz, the author of “The Paradox of Choice” (2004), who often cites the jam study in antimarket polemics lamenting the abundance of consumer choice. In Schwartz’s ideal world, stores wouldn’t offer such ridiculous, brain-­taxing plenitude. Who needs two dozen types of jam?

“The study hardly seems mine anymore, now that it has received so much attention and been described in so many different ways,” Iyengar, a professor at Columbia Business School, writes in “The Art of Choosing.” “From the various versions people have heard and passed on,” she adds, “a refrain has emerged: More is less. That is, more choice leads to less satisfaction or fulfillment or happiness.”

Now Iyengar is having her own say about the jam experiment and the many other puzzles and paradoxes of choice. More choice is not always better, she suggests, but neither is less. The optimal amount of choice lies somewhere in between infinity and very little, and that optimum depends on context and culture. “In practice, people can cope with larger assortments than research on our basic cognitive limitations might suggest,” Iyengar writes. “After all, visiting the cereal aisle doesn’t usually give shoppers a nervous breakdown.”

SEARCH AMAZON USING THIS SEARCH BOX:
Purchases made through this search function benefit this site, at no extra cost to you.
We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites.