Tuesday, May 24, 2011

More on the Kanazawa Scandal at LSE

Satoshi Kanazawa

Further to our discussion last week, I emailed Add Health about the nature of the data that Kanazawa used in his scandalous post seeking explanation of the alleged finding that black women are "objectively" less attractive than women of other races. They sent me a statement this morning, which has obviously been sent to many others who enquired. Here are some excerpts:

"The data Kanazawa used for his research were drawn from the National Longitudinal Study of Adolescent Health (Add Health), a congressionally-mandated study funded by the U.S. National Institutes of Health. Add Health data are available in two forms: a “public use” data set, which includes data from a subset of participants, and a “contractual” or “restricted-use” data set, which includes the full set of variables and participants. The “restricted-use” data are available to researchers who have appropriate research credentials (e.g., post-graduate degree) and an Institutional Review Board in their research institution that ensures their use of data security procedures required by Add Health to protect data and participant privacy and confidentiality. Kanazawa applied for and was granted access to these restricted data, as have thousands of other researchers . . .
"Kanazawa based his blog post on data derived from interviewer ratings of the respondents that were recorded confidentially after the interview was completed and the interviewer had left the interview setting. It is a widely-used and accepted survey practice for interviewers and researchers to include such post-survey completion remarks. These remarks provide both an additional observation about the respondent and data on the context of the interview for researchers to assess data quality. . .
"Interviewer ratings of respondent attractiveness represent a subjective “societal” perception of the respondent’s attractiveness. We included these items because there is a long line of research evidence that indicates that perceived attractiveness is related to important health and social outcomes, including access to health care, health education and instruction, job search, promotions, academic achievement, and social success in friendship and marriage. For example, males who are rated more highly attractive tend to have higher wages, shorter periods of unemployment, and greater success in the job market . . .

"Because the interviewer’s perception is subjective, researchers need to account for the characteristics and life experiences of the interviewer in interpreting their ratings. A wealth of research on perceived attractiveness (that is, as perceived by others, not oneself) has shown that such ratings vary according to the characteristics of the rater. For example, a male interviewer might rate a female’s attractiveness according to different criteria than a female interviewer rating the same female’s attractiveness. Other interviewer characteristics that are important to take into account are age, race, ethnicity, education, geographic location, and life experiences, in general."

Though some of us may have sussed out the main points, especially those contained in the final paragraph above, this statement clearly demonstrates the irresponsibility of Santoshi Kanazawa's "research". As somebody who was granted access to the restricted-use data-set, he would have been aware of the nature of the attractiveness rating. Yet, he gave no indication of this in his blog post.

Add Health Director, Dr. Kathleen Mullan Harris, a professor of sociology at UNC, Chapel Hill drove the point home in an interview with NPR, quoted in the email sent to me: "He's mischaracterizing the objectiveness of the data — that's wrong. It's subjective. The interviewers' data is subjective."

It is not unreasonable to ask that LSE investigate Kanazawa, and take appropriate action. I leave to others who are more experienced in this kind of action how this request could be prepared and communicated to LSE.

12 comments:

I suppose this particular post of Kanazawa'a has struck a chord because it deals with race, but there are many other ridiculous posts by this guy (http://www.psychologytoday.com/blog/the-scientific-fundamentalist). He conforms beautifully to the worst stereotypes of evolutionary psychology.

WHAT??? Is this for real??? I can't believe that the NIH funds this. And btw am I the only one who finds the idea of adult interviewers rating the attractiveness of adolescent interviewees unbelievably creepy and inappropriate?

In response to tomkow, Kanazawa's blog posts misrepresented the nature of the data, in an astonishingly cavalier way--possibly deliberately even. And he did this to arrive at a racist conclusion. LSE should look into the quality of his "research."

As for whether he enjoys the protections of tenure, I have no idea. Whether he does or not, I didn't say that he should be fired, nor do I think he should be on the basis of what I know. But it doesn't seem inappropriate to me for him to be at risk for promotions etc.

Gabriele: You may be right about the inappropriate or creepy for interviewers to be rating these kids (for that's what they are) for attractiveness. But the Add Health statement that I quote gives a rationale for the attractiveness rating. I don't find that rationale stupid. It would be a significant finding if adolescents thought to be attractive went on to higher salaries and better jobs than others. And it wouldn't invalidate the significance of this finding if the ratings were subjective.

One point that puzzles me now. Was the race of the interviewers recorded. Kanazawa talks about racial biases in the judgements of interviewers. So why does he call the ratings "objective"? Why doesn't he take perceiver-bias into account? It's staring him in the face in his own bar charts.

It would be a significant finding if adolescents thought to be attractive went on to higher salaries and better jobs than others.

In what sense would it be a significant finding? From a practical point of view, what would be the policy implications of that finding? And from a theoretical point of view, would anyone be surprised if,as it is likely, it turns out that people who conform to the standards of attractiveness have better chances than those who don't?

Consider this pearl of wisdom for example:'We included these items because there is a long line of research evidence that indicates that perceived attractiveness is related to important health and social outcomes, including [...] marriage.' Really? Perceived attractiveness is related to marriage? Don't tell me! And "good looking people" get jobs more easily than "bad looking people"? Noooo! That can't be true!

In any case, even if those findings were in any way interesting, I don't think that any kind of research that leads to interesting findings is thereby ethically permissible.

One last point--Kanazawa's piece and (the portion of) the ADD Health study on which it relies are so deeply methodologically flawed in so many ways that it seems to be a waste of time to even try to spell them all out. Just one example, presumably they used more than one interviewer, so how did they ensure that the attractiveness judgements of different interviewers were comparable and that they were not comparing apples and oranges? Did they check the consistency of the judgements of all interviewers at some point during the study? I very much doubt they did. Moreover, even if they were comparable, isn't a sample of one (or maybe two/three) interviewer(s) likely to be a pretty small and unrepresentative sample of the societal views about attractiveness?

This is such a methodological mess that I don't even know where to start! So, I don't see how these so-called "findings" could be in any way interesting.

The question is whether this is a case of scientific misconduct and, according to the Guardian, LSE is already investigating it as such. As far as I know, tenure (or its UK equivalent) does not protect people who engages in scientific misconduct (and I don't see why it should).

Gabriele, if there is a known variable that's already been shown to have an effect on outcomes, and your study is meant to meant to determine other variables that correlate with outcomes, it makes sense to capture data on that variable. If nothing else, you can then test whether this variable is interacting with other variables you might find more interesting.

My main problem with all this is that this variable whose correlation (or lack thereof) with other variables is supposedly being studied does not stand for any real feature of the (natural, social, whathaveyou) world. There is nothing that the ADD Study is measuring when allegedly measuring the attractiveness of these kids because the measurement process is completely flawed and does not guarantee any consistency between any two different measurements.

Is it because he has shoddy data? If so, then we would just point out that weakness in the research.

Is it because of his wording? "Black women are objectively less attractive." This sentence makes it sound like attractiveness itself is objective. Yes, attractiveness is subject, but we can objectively measure the subjective reaction of a population. If we said, 1000 people were surveyed with the question, "How attractive do you find this person?" and pictures of black women got an average rating 6.5 and other races got an average rating of 7.5, then that would be a more responsible way of saying it.

Are we upset at the conclusions drawn from the data? He suggests that testosterone and genetic mutations are responsible. It's possible that there are cultural influences, for example the glorification of white people in Hollywood movies might make white people more attractive in the general population's eye.

I see lots of problems like the ones above in Kanazawa's research, but I don't think the backlash is because he's a crappy scientist. I think it's really because the suggested conclusion of that research hits a moral nerve with us. We want to believe that there is beauty in all, that all races are equal, and all the arguments we make against Kanazawa, while valid, are post hoc reasons stemming from our moral intuition.

I think it is as dangerous to resist a conclusion based on our moral biases as it is to concoct one based on racial preferences.

An emotional backlash "in the defence of good science" does her no honor. If we are to stay true to the principles of science, then a dispassionate peer review and criticism with the goal of improvement does her a better service.

Hello,We facilitate the provision of independent analysis to support expert testimony, regulatory or legislative engagements. Frequently, this work includes economic, financial and statistical studies of varying data analysis, technical and http://www.fidelityadvisory.com.