A critical look at the SAT and ACT

By
Valerie Strauss

Of all the trials facing high school students, the one that may be the most daunting is taking the SAT and/or the ACT college admissions tests. The results can determine where they attend college, making these exams, and the organizations that own them, very powerful.

In an effort to better understand these tests, I had a long email conversation with Bob Schaeffer, public education director of the National Center for Fair and Open Testing, known as FairTest.

FairText is a non-profit organization that is dedicated to ending what it says are misuses and flaws in standardized testing. It is quoted by many education journalists when they write about standardized testing and college admissions exams. It is, to put it bluntly, the bane of the non-profit College Board, which owns the SAT. I asked the College Board if a representative would engage in a debate with Schaeffer about the SAT and was told that the organization does not consider Schaeffer, or FairTest, a valid critic.

Here is my conversation with Schaeffer. It is long, but it is worth reading.

Q) First let’s talk about the SAT. Why do you object so strongly to its use in college admissions?

A) FairTest has strongly criticized the misuse and overuse of the SAT (and ACT) because independent research demonstrates that the tests are inaccurate, biased, highly susceptible to coaching, and not necessary for making high-quality admissions decisions.

We do not, however, try to "ban" the exams. Rather, FairTest has long been the leader in the movement for test-optional admissions, which allow applicants to choose whether to submit SAT/ACT scores.

As of today, nearly 850 accredited bachelor-degree granting colleges and universities (http://www.fairtest.org/university/optional)-- about one-third of all such institutions -- will admit all or many students without regard to test scores.

The test-optional list now includes 33 of the nation’s top 100 liberal arts colleges, based on the most recent "U.S. News & World Report" rankings. Since "new" versions of the SAT and ACT were introduced in 2005, more than five dozen additional schools have dropped their admissions testing requirements.

What we definitely do seek to eliminate are widespread misuses of the exam, such as requiring minimum scores for admission (a direct violation of the test-makers’ own guidelines for proper use), linking tuition aid solely to performance on a standardized test (e.g. the so-called National Merit Scholarship should really be called the "National Test-Taking Scholarship" since 99% of all competitors are barred from receiving awards based solely on their Preliminary SAT scores), and requiring admissions test scores for jobs, a purpose for which neither the SAT nor ACT was designed. Unfortunately, the testing industry generally turns a blind eye to improper applications of its products.

Q) Is the ACT any better?

A) The ACT is a different test from the SAT, not a better one. Neither exam is as fair or accurate a tool for predicting college performance as is an applicant’s high school record, according to the test-makers themselves.

The ACT has, however, become a more consumer-friendly test in recent years, featuring an optional writing section, which makes the exam shorter and less costly than the SAT with its mandatory writing component, no penalty for guessing, and the ability for students to control whether a college sees a score from a particular test administration. Partially in response to these initiatives, nearly as many members of the high school class of 2009 took the ACT as the SAT.

Q) Is there any test that would be fair for that purpose?

A) It is highly unlikely that any mass-administered standardized test could ever be as good a tool for predicting college performance (the sole purpose of an admissions test) as students’ high school records. An applicant’s transcript includes data from dozens of classroom tests and other assessments, such as essays and science experiments, which provide a much, much deeper and richer picture of academic preparation and motivation than any one-time exam, particularly one that is based largely on filling in multiple-choice bubbles.

Q) So what is the result of coaching on the ACT and the SAT? The College Board says the SAT is not really coachable. I’ve been told scores can rise a few hundred points for some kids who are coached.

A) Many test preparation firms "guarantee" gains of 300 points or more on the three-part SAT (score scale 600 - 2400) and 3 to 4 points on the ACT (scale 4 -36). Improvement of these magnitudes could have a significant impacts on a candidate’s chances for admission at selective colleges and result in thousands of additional dollars in so-called "merit" scholarships, which are often improperly based on test scores.

If these promised score increases are legitimate, they give children from families who can afford $1,000 for a basic course given by Kaplan or Princeton Review, $5,000 for individual SAT tutoring, or even $15,000 or more for an intensive college-prep package another huge leg up in the admissions process.

Of course, the test manufacturers deny that such score changes are possible. That’s no more surprising than tobacco companies stating the cigarette smoking does not cause cancer: it’s in their self-interest.

As long ago as 1955, an annual report by the College Board, the SAT’s sponsor, concluded, "If the Board’s tests can be regularly beaten through coaching, then the Board is itself discredited."

A number of studies summarized in FairTest’s report "The SAT Coaching Cover-up" demonstrate that good test preparation programs raised scores on the old two-part SAT by 100 points or more (scale of 400 - 1600), significantly more than the testing industry says is possible but less than coaching companies claim.

There’s still no definitive answer to the question, "How much does coaching help?".....

Q) I asked College Board officials if they would take part in a debate about the SAT with you, and they refused. Why do they dislike you so much?

A) It’s always difficult to assess someone else’s motivation. But it appears that the College Board refuses to debate FairTest because they know we have carefully read and digested all their publications as well as independent research about the SAT....

Moreover, FairTest is not uncomfortable sharing data we have gathered about the business side of the College Board demonstrating how profitable marketing college admissions exams has become and how highly paid the company’s executives are (e.g. the President of the College Board receives total compensation in excess of $750,000/year with an expense allowance of an additional $125,000) .

Even worse than declining to debate FairTest, which is, of course, their legal right, the College Board actively seeks to dissuade journalists from interviewing us by refusing to participate in television segments, side-by-side essays, and similar formats if FairTest is involved. Thus, College Board executives try to dictate to education reporters who they should cover and how, trying to become, in effect, the "editor" for coverage of testing-related stories.

Fortunately, this cynical strategy has generally failed. Though a few outlets have been intimidated, most simply go ahead and pursue their stories.....

Since an "all-new, improved" SAT (with tailfins?) was introduced in 2005 more than five dozen schools have dropped their SAT testing requirements for all or many applicants, a clear rejection of the company’s primary product.

Q) Why shouldn’t the College Board’s top officials make a lot of money? It’s a non-profit, but the law allows it.

A) In exchange for their exemption from taxation, non-profit corporations must comply with U.S. Internal Revenue Service limitations on "reasonable compensation" for their executives.

Moreover, applicants to colleges which require either Subject Test or Advanced Placement Exam scores face a monopoly, in which there is no alternative to paying the fees the College Board sets (the market for undergraduate admissions exam is effectively a duopoly with a choice only between the SAT and ACT).

Of course, even in this non-competitive context setting salary levels is more of an "art" than a "science." But paying the head of the College Board more than the presidents of top-tier universities certainly feels excessive. It’s especially annoying to test-takers and their parents who have no choice but to pay ever-escalating fees ....

Q) Of the schools that have dropped the SAT admissions requirement, how many are large schools that get tens of thousands of applications?

A) Many large public universities, such as the University of Arizona, have never required SAT or ACT scores. Others, such as George Mason and Christopher Newport in Virginia and Salisbury in Maryland, have recently eliminated test score requirements for applicants with top-level high school records.

The University of Texas system does not consider SAT or SAT results for in-state students who graduated in the top 10% of their high school classes, a group that now makes up more than 75% of all enrollees at the flagship Austin campus.

A full list of nearly 850 test-optional institutions, including liberal arts colleges, large universities, and specialty schools, is available free online ....

A major new book, "Crossing the Finish Line: Completing College at America’s Public Universities," co-authored by former Princeton President William Bowen, adds important data to the debate about the value of test-scores at large schools. Among the most relevant findings (p. 226):

- “High school grades are a far better incremental predictor of graduation rates than are standard SAT/ACT test scores”
- “Overly heavy reliance on SAT/ACT scores in admitting students can have adverse effects on the diversity of the student bodies enrolled by universities”
- “The strong predictive power of high school GPA holds even when we know little or nothing about the quality of the high school attended.”

This research is already encouraging several large universities to review their admissions testing requirements

Q) A question about ACT. I’ve talked to test prep folks who say that the ACT is fairer than the SAT because it supposedly tests what kids have learned, unlike the SAT. Is this true or not true? If true, why wouldn’t it be useful in helping predict college success?

A) Yes, both the content of the ACT is closer to the material a student has covered in high school (in fact, it is based on a national curriculum survey). But the test-makers’ own research show that the ACT is neither more accurate nor fairer than the SAT in predicting college grades. The most likely reason for the tests’ relative weakness is that no half-day, largely fill-in-the-bubbles exercise can capture key elements of academic success such as motivation, perseverance, work habits and coping skills.

Q) So what are parents and students to make out of all this? Is there something a parent can actually do?

A) College-bound students and their parents should understand that there are now more options than ever in dealing with the standardized testing hurdle for undergraduate admissions. Nearly one-third of all colleges and universities do not require SAT or ACT scores from all or many applicants. The test-optional list includes dozens of top ranked liberal arts schools, many large public campuses, and specialty schools focused on the arts, religion, music, design and more.

The existence of so many test-optional alternatives can significantly reduce the pressure and anxiety about scoring high on one-shot exams. Students can, in essence, opt-out of the test-prep game by focusing on these schools which recognize "you are more than a score."

I generally support Fair Test as I think any watchdog over the College Board is a good idea. However, Fair Test does not tell the whole story. While HS grades do a slightly better job of predicting a student's first year college grades than either the SAT or ACT, either of those tests in combination with grades does a better job than grades alone. A recent study by the state college system in NY suggested that SATs are a better measure than HSGPA for predicting college graduation rates.
So the question for Fair Test really is why would they advocate making optional a tool that will help colleges better assess students' prospective performance without proposing a replacement?

Someone should propose to Mr. Schaeffer that if his organization is really serious about its name, Fair Test, that it should be spending some of the money it collects on developing a fairer test rather than merely throwing stones. I for one would love to see Fair Test involved in promoting and developing achievement type tests

I wish you had asked about SAT Subject Tests and AP tests. Those, to me, seem to be very appropriate tests.

I never understand how these disingenuous folks can suggest that the high school record is a better judge when there is so much variation in the level and quality of instruction, between states, schools, and teachers within a single school. There is no way to tell that a B from Mrs. Brown in 12th grade English is like an A from Mrs. Smith in the same class, to say nothing of the vast differences that exist between different schools and different districts.

The so called research findings are simply excuses.
- “High school grades are a far better incremental predictor of graduation rates than are standard SAT/ACT test scores”
Note that this result was not found for the most selective institutions. In addition, the research failed to take into account the fact that students at selective school A have higher SAT scores than less selective school B. Thus, the predictive power of the exam is reduced when you are looking at the difference between a 790 and a 700 on the math section. But the difference between a 700 and 300 student is HUGE. I cannot believe a student with a 300 math or verbal score on the SAT could survive in college. Is that what he is asserting?

Also, I can believe that people who took easy classes in high school to maximize their GPA will take easy majors in college. Many intelligent students go into difficult fields where graduation is actually a challenge. The study does not control for major or difficulty of school on this statistic. So, if I am an A student with a 400 SAT Verbal and I get into easy school U1, I am going to have an easier time of graduating than a B+ student with a 800 SAT Verbal getting into challenging school U2.

- “The strong predictive power of high school GPA holds even when we know little or nothing about the quality of the high school attended.”
I find that hard to believe, when A students at many schools are unable to take the lowest level college Math or English courses. In the aggregate, the stats may work, but that means nothing when considering an individual. If you have a choice of a student that got an A in AP Chemistry but a 1 on the AP test and a student that received a B in the class with a different teacher but a 5 on the test, which would you choose? The practical implementation is obvious.

The SAT and ACT is all about context. That is they can provide a context of a person's grades. Colleges are lazy. These tests are only a starting point. Even though SAT II Subject tests provide some additional evidence of background for college,they still present, at the very least, an incomplete picture.
I remember back in college, those with high SAT scores who could not do fractions for example, or had a demonstrated inability to write complete, effective sentences.

This is about college sloth. With the internet, there are numerous interactive opportunities to evaluate students, besides, the staged, manipulated applications process.

If you are a top student, ask the question is the college worthy of you? Their application process can provide revealing evidence.

Probably would have been good to note somewhere that the same company that owns Kaplan test prep owns the Washington Post. (And, the wildly successful income generation of the test prep is arguably helping to prop up the declining newspaper revenues.)

Always good to err on the side of caution with noting these conflicts of interests.

Glad to see such a thorough, and prominent, conversation with Fair Test!

Taking dozens of practice tests -- whether through paid "coaching" or for FREE, through the hundreds of free practice tests offered by the SAT's organization itself -- does improve your scores, by as much as 100 points.

So what? Such preparation is: (1) actual learning of how to perform under pressure, a skill that isn't well taught in many high schools, is very necessary to college performance, and therefore is indeed predictive of college success; (2) actual substantive learning of vocabulary, basic math, and problem solving, which again are important skills to learn and aren't well taught in many high schools; and (3) demonstrates that the kid who takes all these practice tests (whether paid or free) is a self-starter who will be able to run his/her own life in college much better than the kid who didn't even bother to take free practice tests before taking the SAT.

Kids who succeed in college (and life) are the kinds of kids who practice hard for the SAT. That's the important variable here. Comparing the success of kids who prepare for the SAT versus those who don't bother to do so -- again, remember, most of the prep materials are FREE -- is ridiculous. FairTest and similar organizations never acknoledge this fundamental fact.

Someone should propose to Mr. Schaeffer that if his organization is really serious about its name, Fair Test, that it should be spending some of the money it collects on developing a fairer test rather than merely throwing stones. I for one would love to see Fair Test involved in promoting and developing achievement type tests.

-----------------------

This is a terrible idea for two reasons:

1) Offering one's own test completely ruins any independent credibility in assessing the use and value of testing. If FairTest starts promoting their own test, then any criticism of the SAT becomes marketing and not objective analysis.

2) FairTest's point is that there is no good, fair way to test students nationally on the mass scale. Any test will be inherently unfair and unreliable. Creating their own test would be patently hypocritical.

"This [having FairTest release its own test] is a terrible idea for two reasons:
1) Offering one's own test completely ruins any independent credibility in assessing the use and value of testing. If FairTest starts promoting their own test, then any criticism of the SAT becomes marketing and not objective analysis. 2) FairTest's point is that there is no good, fair way to test students nationally on the mass scale. Any test will be inherently unfair and unreliable. Creating their own test would be patently hypocritical.
===============

Fair point. But isn't that an admission that FairTest isn't just trying to create a fair test -- it's against any national testing, of any kind?

Colleges don't use the SAT as 100%, or even 50%, of a candidate's qualifications. I know this, since I recruit for an elite college. But we certainly do use it as a part of the picture, and it's very valuable. It's most valuable, in fact, for identifying the LESS advantaged kid, with the middling grades, who tests far better than the rich preppy kid from St. Fancypants Academy. The SAT is great at identifying the "diamond in the rough" kid that we'd otherwise miss by using all the metrics that FairTest so piously touts as alternatives. The SAT is, in short, helping the very kids that FairTest claims it hurts. If a poor inner-city kid scores a 2000 and the St. Fancypants suburban rich kid scores a 2100, we think the poor kid is the one to call back first. Even a 300 point gap, or more, would still be seen as a positive for the poor kid. C'mon, we're not robots, and we're not stupid about the SAT.

The FairTest people are assuming that we discriminate in ways that we don't, that we act as if a 5 point difference on the SAT is a life-changer, and that the SAT hurts people it actually helps. FairTest may have good intentions but it's hurting the very people it claims to be defending. Unless, as I suspect, FairTest's real agenda is to wipe out all standardized testing because it doesn't like the idea of testing that exposes intelligence differences between kids.

Easy way to check: does FairTest endorse any major standardized test? No. Does it work to help others improve or create a more "fair" test? No. Does it do anything constructive to help create objective testing, so that kids from differing backgrounds can be evaluated with an equal instrument? No. QED.

Here's one example of how Schaeffer exploits the truth. There are approximately 1,500 schools that most would consider "traditional colleges"; these are your 4-year, not-for-profit domestic colleges. Of the 850 institutions that Schaeffer quotes, the vast majority are not in this group; they are bible schools, nursing schools, for-profit schools, music schools, etc. Of the 1,500 traditional colleges, only ~60-70 are test optional, and almost all of these are small liberal arts colleges in the NE. So the percentage of students in traditional colleges that go to non-test optional schools is in the mid-90%. Test optional is not a big trend. Too bad the "reporter" didn't include this or other facts in the post