Professor Heather Williams, chair of the Athletic Committee for 2008-2009, kindly replied to my request to know more about the details of their statistical analysis (pdf).

As chair of the committee, I am responsible for the statistical analyses. The design of the analyses was the responsibility of the committee, and several of us are statistically competent. The analyses themselves were run by the Provost’s office, and although you are certainly free to ask Chris Winters about them, I would imagine that he’ll be referring all questions to me. The report will not describe all of the results in detail, but will provide enough information about how the analyses were set up and the results to let the readers understand exactly what we did.

A quick summary, we ran a number of simple linear models (and logistic regressions, where appropriate to the question) that included a variety of variables that seemed likely to have affects on academic performance: reader rating, gender, class (fr., so., etc.), a proxy for socioeconomic status, and sport category (high/low profile), as well as a number of interaction terms.

I’m sure that, as an economist who is into numerical analyses, you would prefer to have more information, but I wouldn’t be comfortable about giving any additional details beyond this summary of how the analyses were done. The more details we give, the more people want to know, and it’s important not to make it possible for anyone (including us) to gain too much information about individual athletes.

Perfectly reasonable! I still think that the Committee ought to share the regression results (which are available in an Appendix on file in the office of the Dean of the Faculty) with the entire community, but this is a minor quibble. The Report is high quality and represents the best of the Williams tradition of faculty governance and transparency. Kudos to all involved!

For those interested, below is the e-mail that I sent to Professor Williams.Read more

This year’s athletic report is much, much shorter than the one I first discussed in 2004 (known as the MacDonald report); this one is just a follow-up. It is only four pages long, so any of you who are really interested should consider reading the whole thing. I have excerpted over 50% of the entire document below.

First, good news:

We find that the gap in academic performance, as judged by grade point average, has narrowed substantially overall and has essentially disappeared for female athletes and for male athletes in low-profile sports. The gap for male athletes in high-profile varsity sports (which we defined as football, ice hockey, basketball, and baseball […]) appears to be narrowing, but persists even after we adjust for 1) academic qualifications prior to enrolling at Williams College, 2) socio-economic status, and 3) the individual’s year (e.g. sophomore, senior). Thus academic under-performance by male varsity athletes in high-profile sports continues, and cannot be attributed to academic credentials prior to Williams or to socioeconomic status.

Kudos to Professor Heather Williams, chair of the Athletics Committee for her decision to make public their latest report. (Permanent archive here as well.) The conclusions were described in an earlier Recordarticle.

The Athletics Committee conducted a report on academic performance of varsity athletes versus non-athletes to be completed and submitted to the faculty in May. The first study conducted since Michael MacDonald, former chair of the committee, released a report in 2001, it found that the overall gap in academic performance has been halved, and that gap has been eliminated for females when averaged across all sports. Despite this progress, a considerable discrepancy does remain for high-profile male athletes in sports such as football, basketball, baseball and ice hockey.

In the Spring of 2002, an ad-hoc committee chaired by Michael MacDonald reported on its review of the role that varsity athletics played at Williams College.

We have been referring to that document as the Report on Varsity Athletics, but Williams et al use the term “MacDonald report.” I will do the same. It is beyond pathetic that Director of Public Affairs Jim Kolesar refused my request to make that report public. It is only thanks to the kindness of Professor MacDonald that any of us were allowed to read it.

The history of the MacDonald report is fairly obvious, if always unstated. Morty arrived at Williams and was upset by the magnitude of the admissions preferences given to athletes and the (resulting?) disengagement of many athletes from the intellectual life of the College. And, being a smart president, he knew that the way to change things was to appoint a committee of like-minded souls who would recommend the changes that he wanted.

The 2009 Report continues:

Shortly thereafter, Bowen and Levin published “Reclaiming the Game”, a book that reviewed varsity athletics in three conferences (Ivy League, NESCAC, and UAA) whose institutions place a high value on academics. Both the book and the MacDonald report concluded that varsity athletics (within NESCAC and at Williams) posed problems for some aspects of the academic mission of colleges such as Williams.

Indeed they did. If you haven’t read the MacDonald report yet, you ought to. It is superb and compelling. But note that the reference to “Reclaiming the Game” is somewhat curious. The book that had a major influence on the MacDonald report was “The Game of Life” by Shulman and Bowen. Both books are co-authored by Williams Bowen and make almost identical points: elite colleges put more emphasis on athletics than they used to; this is a bad thing.

This is highly misleading! Did the Committee members not read my criticism of the MacDonald report? I demonstrated, fairly conclusively, that there was no evidence for under-performance by Williams athletes, once we account for their high school academic credentials, despite the fact that the Committee went out of its way to look for such underperformance and even spun its results to make athletes look bad. The MacDonald report itself states that:

The grades of our athletes are lower than the grades of our non-athletes, which should not surprise us. They are often weaker students when they enter Williams, and they commit much time and effort to their sports. It is predictable that they would be weaker students in Williams, suggesting a second question: do athletes underperform academically, controlling for their academic ratings at the time they were admitted? The data is mixed, but suggests on balance that our athletes achieve about the same grades as non-athletes with similar academic ratings.

Day is night. Ignorance is strength. And varsity athletes underperform academically.

Isn’t this pathetic? Let me highlight the conflict again.

MacDonald report: “[O]ur athletes achieve about the same grades as non-athletes with similar academic ratings.”

Now, to be fair to Professor Williams and the rest of the committee, the “they” in their sentence refers to both the MacDonald report and to Reclaiming the Game, and it is true that Bowen and Levin conclude that “recruited athletes “underperform:” they do even less well academically than predicted by their test scores and high school grades.”

But this has always been a highly suspect claim (to me). First, it was clear from the start that Bowen, at least, wanted to find exactly this result. He thinks (has always thought?) that elite colleges should be more like Cal Tech. Athletic excellence should not matter to admissions. Bowen, who does not have a reputation as the world’s greatest statistician and almost certainly did not do any of the empirical work himself found exactly what he wanted to find. It could be true, of course, but I would like to see some independent verification.

Second, the claim of underperformance of athletes was similar to the underperformance claims made about African American students in Bowen’s “The Shape of the River.” But how could both sorts of underperformance be true at the same time? The vast majority of students in the bottom 20% of the SAT distribution are either athletes or African Americans. (The Shape of the River ignores Hispanics.) It could be true that one group does less well than the other, but how could both groups underperform given that they make up the vast majority of the population at those SAT levels? I think that there was some very sloppy regressions run to generate these results.

The 2009 Report continues:

b) varsity athletes tended to cluster in certain majors and c) some faculty members felt that lack of intellectual engagement of varsity athletes posed problems for their classes. The purpose of this report is to re-examine some of these findings, seven years later.

Fair enough. The MacDonald report did, indeed, note the clustering (mostly within Division II) and report accurately about the concerns of these faculty. But there is no evidence that this clustering was unusual given their academic credentials. Are their a lot of football playing math majors? Probably not. But this has nothing to do with football per se. The main cause is that there are not a lot of math majors with academic ratings below 3.

Side note: This Report mentions that “The statistical analyses used in preparing this report are summarized in an appendix that is not part of the report, but will be placed on file in the Dean of Faculty’s office.” Why not make it public too? Would an alum (like me) be prevented from viewing the report if he showed up in Hopkins Hall, say during reunions? Academic institutions should be as transparent as possible. There is no plausible reason for keeping the appendix a secret.

In any event, there is much of value in the Report. Alas, it will take me a few days to wade through it all. Stand by for more commentary.