Saturday, December 31, 2016

Who is responsible when an article gets misread?

How much of the responsibility for understanding lies with the writer of an article, and how much with the reader? This is not an easy question to answer. Obviously both sides bear some responsibility. There are articles so baroque and circuitous that to get the point would require an unreasonable amount of time and effort to parse, even for the smartest reader. And there are readers who skim articles so lazily that even the simplest and most clearly written points are lost. Most cases fall somewhere in between. And the fact that writers don't usually get to write their headlines complicates the issue.

See what you think about this one. The other day, Susan Dynarski wrote an op-ed in the New York Times criticizing school vouchers (a subject I've written about myself). Dynarski opens with the observation that economists are generally less supportive of vouchers than they are of most free-market policies:

But economists are far less optimistic about what an unfettered market can achieve in education. Only a third of economists on the Chicago panel agreed that students would be better off if they all had access to vouchers to use at any private (or public) school of their choice.

Here's the actual poll:

As you can see, the modal economist opinion is uncertain about whether vouchers would improve educational quality, while the median is between "uncertain" and "agree". This clearly supports Dynarski's statement that economists are "far less optimistic about vouchers than about Uber and Lyft.

The headline of the article (which Dynarski of course did not write) might overstate the case a little bit: "Free Market for Education? Economists Generally Don’t Buy It". Whether the IGM survey shows that economists "generally don't buy" vouchers depends on what you think "don't buy" and "generally" mean. It's a little click-bait-y, like most headlines, but in my opinion not too bad.

By leaving it at “only a third of economists support vouchers”, the article implies that there is an economic consensus against the policy. Heck, it more than implies it – its title is “Free Market For Education: Economists Generally Don’t Buy It”. But its own source suggests that, of economists who have an opinion, a large majority are pro-voucher...

I think this is really poor journalistic practice and implies the opinion of the nation’s economists to be the opposite of what it really is. I hope the Times prints a correction.

A correction!! Of course no correction will be printed, because no incorrect statements were made. Dynarski said that economists are "far less optimistic" about vouchers than about Uber/Lyft, and this is true. She also reported close to the correct percentage of economists who said they supported the policy in the IGM poll ("a third" for 36%).

Scott is upset because Dynarski left out other information he considered pertinent - i.e., the breakdown between economists who were "uncertain" and those who "disagree". Scott thinks that information is pertinent because he thinks the article is trying to argue that most economists think vouchers are bad.

If Dynarski were in fact trying to make that case, then yes, it would have been misleading to omit the breakdown between "uncertain" and "disagree". But she wasn't. In fact, her article was arguing that economists tend to have reservations about vouchers. And she supports her case well with data.

This is a special kind of straw man fallacy. Straw manning is where you present a caricature of your opponent's argument. But there's a particularly insidious kind of straw man where you characerize someone's arguments correctly, but get their thesis wrong. You misread someone's argument, and then criticize them for failing to support your misreading. Other examples of this fallacy might be:

1. You write an article citing Autor et al. to show that the costs of trade can be very high. Someone else says "This doesn't prove autarky is better than free trade!" But of course, you weren't trying to prove that.

2. You write an article arguing that solar is cost-competitive with fossil fuels by pointing out that solar power is expanding rapidly. Someone else says "Solar is still a TINY fraction of global generating capacity!" But of course, you weren't trying to refute that.

I think Scott is doing this with respect to Dynarski's article. To be fair, his misreading was somewhat assisted by the headline the NYT put on the piece. But once he was reminded of the fact that the headline wasn't Dynarski's, and once he re-read the article itself and realized what its actual thesis was, I think he should have muted his criticism.

Instead, he doubled down. He argued that most reasonable people, reading the article, would think it was arguing that economists are mostly against vouchers. But his justification for this continues to rely very heavily on the wording of the headline:

Second, the article uses economists “not buying it” as a segue into a description of why economic theory says school choice could be a bad idea...

In the face of all of this, the New York Times reports the field’s opinion as “Free Market In Education: Economists Generally Don’t Buy It”.

On Twitter, he said: "the actual article is more misleading than the headline." But he appears to say this because he takes the headline - or, more accurately, his reading of it - as defining the thesis that Dynarski is then obligated to defend (when in fact she wrote the piece long before a headline was assigned to it). When he finds that Dynarski doesn't support his reading of a headline she didn't write, it is her article, not the headline, that he calls "misleading".

Of course, the fault here is partly that of the NYT, who used a headline that focused only on one part of Dynarski's article and overstated that part. It's a little harsh for me to say "Come on, man, you should know an article isn't about what its headline says it's about!" Misleading headlines are a problem, it's absolutely true. But after learning that Dynarski didn't write the headline, I think Scott should have been able to then read the article on its own, and go back evaluate the arguments Dynarski actually makes. It's the refusal to do this that seems to me to constitute a straw-man fallacy.

Anyway, one last point: I think Dynarski is actually wrong that economists are more wary of vouchers than other free-market policies. Yes, economists in general are probably wary of voucher schemes. But they're also a lot more favorable to government intervention in a variety of cases than Dynarski claims. Klein and Stern (2006) have some very broad survey data (much broader than IGM). They find that 67.1% of economists support "government production of schooling" at the k-12 level, with 14.4% uncertain and 17.4% opposed. But they also record strong support for a variety of other interventionist policies, such as income redistribution, various types of regulation, and stabilization policy. On many of these issues, economists are more interventionist than the general public! So I think if Dynarski makes a mistake, it's to characterize economists as being generally pro-free-market. Their ambivalence about vouchers doesn't look very exceptional.

28 comments:

Do you think using the word "only" in "Only a third of economists on the Chicago panel agreed ..." is misleading if you don't also mention that 37% are unsure? I think the article would have been more clear if the full distribution was shown.

The last point in Noah's piece is the key there. The writer is assuming that economists are generally strongly pro-market and characterizing their position on this one issue as an exception. So high levels of uncertainty add to her point just as much as opposition does.

Let's say a poll showed that only 30% of football fans reported that they were definitely going to watch the Super Bowl this year. Even if another 60% say they might, I'd take that as a sign that this year's matchup wasn't appealing (I have no idea what normal would be--which improves the analogy).

In that case, the NYT author avoids misleading readers with shared priors, while possibly misleading other readers.

As a naive reader, I interpreted the author's phrasing as presenting one side of a binary statement. For example, if all but one panelist declined to respond, and the lone response disagreed, would you promote clarity by writing "only one panelist disagreed with the proposition"? Keep in mind that the writer did not provide further context to suggest delineations beyond agree/disagree.

The Op-Ed is misleading because the argument depends on reporting only part of the data. The author could have also written, "Only 19 percent of economists believe vouchers would make students worse off" which is also a true statement. It is the author's responsibility to be clear. Selectively reporting the data is troubling.

Note also the second, clearer follow-up survey the IGM conducted: http://www.igmchicago.org/surveys/school-vouchers, which makes economist support for vouchers more clear and probably should have been used instead.

I read the NYT article before reading Scott's post and was mislead just as he suggested. I did not click through to the survey and simply assumed that 40 something percent of economists opposed vouchers (with a smaller percentage uncertain). I think anyone who didn't click through and had no prior knowledge would read it in the same way.

If that is true the author has to take responsibility especially since being much clearer would have been very easy ("as opposed to 19% who disagreed").

While you might not want to take Scott's bet, you should ask a few of your friends or family to read the article and ask them what they think it means about economists' views on vouchers.

You should do this because you are wrong and being wrong and finding out about it is a lot more interesting than being right.

You all seem to be making a jump and equating "all economists" with the "panel of economists the University of Chicago considers to be leading economists." That sleight of hand is a far more serious intellectual error than anything Susan Dynarski might have done.

It's called poor writing if the reader can't infer your thesis after reading half a page. John Cochrane would ashamed. It took me three tries before I could tell whether the OP was anti voucher or just cautioning. I hardly fault the critic from inferring the thesis from the title

While I largely agree with Smith here, I think part of what led to Alexander's frustration is that the New York Times, as befits its sobriquet the "Gray Lady", seems to encourage its writers to use an understated and courteous style. Thus, when you read in the New York Times "economists are far less optimistic about x than y", rather than just literally reading it as saying that the polled support for y>x it seems reasonable (though in this context it would be wrong) to interpret that as saying that economists generally think x is a bad idea.

I'm pretty sure when you start disingenuously mocking Scott Alexander, who is unambiguously the most intellectually honest person I read online, more of your sludge bounces back on you than lands on him. To the point you raised: it's clear that when a majority of people misunderstand your newspaper article designed to inform, the fault lies not with your audience but with you.

When I see this type of argument in my line of work, I have a lot of fun with it - pointing out that my opponent is doing a great job refuting a point I wasn't making and don't need to make. I'm never sure whether (1) my opponent is making a straw man of my argument; (2) I wrote the original argument in a confusing way that yielded this response; or (3) my opponent is so used to seeing a somewhat different (usually a more sweeping) argument that they didn't notice that I wasn't saying the same thing they anticipated. I think (3) is a strong suspect because of confirmation bias, so I tend to say, "let me be clearer" instead of "she's twisting my argument!" as often as possible. Of course, that is also a slippery way that many people try to escape good counterpoints, as I think you've mentioned on this blog before: changing the argument in response to criticism, to be a moving target and shift all the blame to someone else for their misunderstanding or mischaracterization. So it's a tough question! At least, in my line of work, I get to write my own headings, and someone gets to make a final pronouncement as to whose argument worked better.

Picking and choosing data is an insidious and hugely damaging problem in academia and science reporting. Scott Alexander is right to try to raise the bar even if this may not have been the worst example of it.

It's inefficient and unrealistic to try to put more responsibility on readers when it's so much easier for authors having full papers in front of them to paint a more complete picture of the data than for readers to go find original sources when they read newspapers.

Also Scott offered to resolve this empirically with odds in your favor. Why wouldn't you take that bet?

Alexander is being ingenuous. The Dynarski op-ed simply argues that economists have no consensus on the value of vouchers. In contrast, they do have a consensus on a number of other issues both in favor of free market operations and more skeptical of free market benefits.

Alexander argues that the lack of consensus can be read as 2/3 approval just as easily as 2/3 disapproval and therefore .... Well I'm not sure of what is argument is. Does an ambiguous 2/3 approval count as a consensus? He seems to be making that claim. On the other hand, there is an ambiguous disapproval. Despite this, as far as he is concerned, the argument is settled. Either that or he missed the entire point of the op-ed.

To me, the argument is settled. Charters don't help. Vouchers for elementary education make as much sense as vouchers for military defense, but I'm not making an economic argument. (Don't laugh. During the Vietnam War, a lot of people argued for defense vouchers.)

The NYT ought to publish a correction for the headline. The op-ed is fine, nothing wrong with it.

The graph from IGM also looks incorrect, by the way. The actual table of responses shows that of 41 economists, 34% (14) agree, 20% (8) disagree, and 46% (19) were uncertain or did not answer.

But Susan's point was clearly to emphasize "striking a balance between free competition and regulation". Not only does the raw data support her here, but it is especially informative to read the comments offered by 28 of the IGM panellists, the overwhelming majority of which seem to support Susan's position, regardless of how they voted on the question. In fact, of the 12 comments offered by those who voted in favor of vouchers, 7 of them independently chose to highlight either that some students would not be helped or that some might even be harmed.

Ultimately, it seems there is a consensus here, it's just not one that is in favor of either unfettered free markets or complete governmental monopoly command and control.

A further flaw in Alexander's reasoning is equating uncertainty with an absence of opinion. He states, ". . . of economists who have an opinion, a large majority are pro-voucher." By this he apparently means to dismiss the plurality of respondents who ticked the box labeled “Uncertain.” Of course, this response can be interpreted in different ways, but given that "No Opinion" was also reported (with 0% of respondents), I would understand "Uncertain" in this case to mean that the respondent had considered the question and reached the conclusion that a voucher program would be of negligible or ambiguous benefit. (Reading the comments of the respondents confirms this intuition.) This is plainly not the same as having no opinion.

I'm surprised you didn't employ economic reasoning here, similar questions are asked in contract law. The Kaldor-Hicks solution is the cheaper information gatherer, in this case it would be on the author to provide the data.

The article is misleading, while not making any literally incorrect statements. I'm surprised you would argue otherwise, given your previous writing on cherry-picking data.

"I don't buy it" means "I don't accept that it is true," not "I have concluded that it is false." Economists are supposedly scientists and they not supposed to accept the truth of something without data. If economists "don't buy" a hypothesis it means that they have not seen data that persuasively supports it. It's not the Times' fault that readers like Alexander don't understand the scientific method. There's only so much ignorance that you can dispel in a single headline.

The problem with Dynarski's piece is while she's written it in such a way that while each individual statement is true, or at least valid, the way it's structured forms a "pseudo-argument" that doesn't actually claim that most economists are opposed to vouchers but nevertheless strongly implies that it's making such a claim.

It's not just the title that gives this impression - she sets off the piece by saying "You might think economists would agree with this approach..." - which, of course, implies that she's concluding "... but they don't." However, that's actually a strawman - what she's provided is a (valid) argument for why economists who otherwise support free markets might still support government regulation in the case of education. The 30% statistic, though, is positioned in such a way as to imply "not only MIGHT they agree, but in fact most of them DO agree" - however, that's not what the stat actually says.

Here's an example of what I think she's doing - (stats made up)"We all know monkeys love fruit - so you might think that monkeys love grapes. However, in a series of studies, while 90% of monkeys preferred bananas to carrots, only 30% of monkeys preferred grapes to oranges. Not all fruits are created equal - some are poisonous, in fact, and monkeys are well aware of this. Many berries are poisonous, and grapes bear a strong resemblance to certain poison berries, so an aversion to grapes could be quite sensible from an evolutionary perspective."So, did I claim that monkeys don't, in fact, like grapes? I suspect most readers would say I did - but, in fact, I didn't say any such thing.

I agree that there was some overreaction to the headline and that Dynarski's statement was mostly true (if slightly misleading as a plurality of economists seem to hold positive views of vouchers), but it is worth pointing out another later survey from Chicago had even more economists viewing vouchers favorably: http://www.igmchicago.org/surveys/school-vouchers (44% of economists held positive views of vouchers opposed to 6% negative. Still lots of uncertainty put thats ok and on par with the research on the effectiveness of vouchers on students using them). As for Noah's posted survey, does the "government production of education" refer to the government owernship and operation of K-12 schools? If so, the results of the two surveys are at odds. I wonder though, if the economists were offered any alternative where the government might assist the poor and disabled afford a private education... such as through vouchers. If not, perhaps the results aren't at odds, the survey question was simply incomplete.