Because it doesn't measure IQ. It is used that way. And it was developed
from the army IQ test. But even the College Board will refuse to say that this
is an intelligence test. And I'd love to see them say it. I'd love to see them
say anything because then you can attack it. But there's this kind of mushy
response that when you work your way through it, there's sort of nothing
left--'Well, it has a slight predictive validity to freshman year grades in
college.' We spend a 100 million dollars a year for that? You know--your grades
in high school predict college grades better than this and we didn't have to
spend anything. They say the SAT provides a common yardstick for comparing grades at
different schools.

Right, that's where all of the anti-test people, I think, are wrong. And
where the testing folks are right. You do need a common yardstick. You do need
some way to judge an A at this school or this teacher versus an A at this
school or this teacher. But there are lots of common yardsticks. Again, you
could use blood type. You could use height. Anything is a common yardstick.
What you have to say is, fine, it's common. But it is useful? And there are
lots of tests that are more useful than the SAT that are also common.Such as?

Well, for instance, the Advanced Placement Tests. They are rigorous.
They're difficult. There are lots of them. You can say, my interest is history.
And so I'm going to take a history Advanced Placement test. And I'm going to
get--I want kids to be rigorous. I want curricula be rigorous. But I want them
to be not one-size-fits-all and not mindless. Let's have kids studying hard but
let's have them studying something useful, hard. You say the SAT doesn't measure intelligence. What does it
measure?

The SAT is said to predict freshman year grades in college, a little. And
it does. It measures it a little. Almost anything you do, including family
income, will measure freshman year grades a little. But the point is that it
doesn't measure intelligence. It doesn't measure anything that's worth a 100
million dollars a year prepping for it.

Are you saying the SAT is a guessing game?

Part of the SAT is a guessing game. Fair Test's position is reasonably new
on it. We've never said that the SAT is not measuring something meaningful.
There's a little part of meaning, there's a little part of background, there's
a little part of schooling. But there's a lot of test-wiseness. There's a lot
of 'how shrewdly you can play the game?' There's a lot that can be taught in
coaching courses that has nothing to do with any of the skills you need to
succeed in college or in life.Wayne Camara, head of the College Board's Office of Research, says the
SAT is a measure of verbal and mathematical reasoning ability.

Well, Wayne Camara, who I know is a decent person--maybe, he believes that.
But if you talk to representatives of tobacco companies, they tell you their
products are good and healthy and don't lead to diseases. And auto
manufacturers have told you that their cars don't burst into flames. And back
at the beginning of the century, you know, stockyard owners claimed that they
were slaughtering the animals in a clean way. I think we've learned not to
trust those who profit from manufacturing products, as the sole source of
information.

Yes, one of the things the SAT is measuring in small part, are those
skills. But it's measuring a whole lot of other things. And when you use the
SAT as the major factor--or worse, the sole factor--to make high stakes
decisions, to define what is merit, you're relying not on what most people
think of is merit. But on these very trainable skills that coaching courses and
others help people learn.What does the SAT predict?

The sole scientific claim of the SAT is its capacity to predict first year
grades. According to the technical studies done by the Educational Testing
Service and College Board, the SAT predicts about one factor in six--one sixth
of the difference between two kids' first-year grades. The predictive value
declines after that--looking at four year grades or graduation rates. So even
the test makers agree that five out of six parts of whatever it takes to
predict how well you're going to do in your freshman year, is not their test.

It does correlate extremely highly with an IQ test. It was developed from
the army IQ test...

That's part of the seedy under side of the SAT. The SAT was originally
developed by straight out racists--eugenicists, people who thought my
forbearers--not just people of color--were imbeciles and shouldn't be allowed
in their country because they didn't know the language and couldn't score high
on their test. I wouldn't suggest the current people who run those companies
share those kinds of ugly views. But it's a self-reinforcing notion of defining
intelligence as that which whatever the dominant group in society has. Ends up
giving that group higher scores and lower scores. The fact that test scores
correlate with test scores is rather meaningless. The tests are measuring the
same set of factors. What's more important is whether the test accurately
predicts how well you're going to do.Would you say that we are the only country in the world that administers
a national IQ test?

Well, despite the efforts of the Educational Testing Service--which is a
global corporation with nearly about half a billion dollars in total
revenues--the US still is the major country that administers a test like this
across the board to college bound seniors. If you take the SAT and its
competitor, the ACT--which about 80 percent as many kids take--the vast
majority of college bound kids take those tests. And yes, the SAT in particular
has its roots in IQ testing. Which are at best controversial and, at worst,
quite, quite poor predictors of anything of value.So is it an IQ test?

It's a variant of an IQ-like test. It is set up somewhat differently. It
begs the question of, what is an IQ test measuring? What is intelligence? And
you talk to test makers. And intelligence is what their test makes. And that's
a circular definition. So to the extent that it's measuring the same that an
intelligence test is measuring--then, yes it is. But there's three fallacies
there: That there is such a thing as intelligence--that it can be measured. And
that you can put the measurements on a linear scale. And other, even people who
believe that there is such a construct as intelligence believes that
intelligencee is not one thing but seven or eight or possibly nine different
things. Robert Sternberg at Yale says it's three different things.

At best, the SAT is badly measuring one of those parts of what goes into
intelligence.

What exactly does the SAT measure? You say reasoning and ability
skills.

The SAT measures two areas. It measures developed verbal reasoning, which
are the type of skills that would be measured by reading long reading passages.
For example, in our new test students have an essay where they would read two
contrasting views on a topic. It could be political. It could be in
humanities. It could be in science. And they need to piece together
similarities and differences of the arguments--contrasting views. So that's a
type of analytic thinking and critical thinking skills that are acquired when
you read essays in college. Or the type of scientific or literature work that
you'll encounter in college and in English.

In mathematics, the SAT measures developed mathematical reasoning. So it
shies away from simple computation. As a matter of fact, the SAT of
today--unlike the SAT that you and I probably took--allows and even encourages
students to bring calculators. So it cannot measure simple addition or
division or fractions because those would be incredibly easy with the use of a
calculator. It has to measure reasoning problems, the type that you would
have, in real world applications.

And it also has a number of items that are not multiple choice. Students
have to read the context, understand the mathematical applications involved,
and then generate their own answer.Isn't it an IQ test?

No, it's not an IQ test. It's far from it. Developed reasoning skills
measured on a test like the SAT, will link directly to the, the breadth and the
depth of the curriculum students have been exposed to in school, but also out
of school learning. Students who have read an incredible amount, whether it's
in school assignments or out of school assignments, are more likely to do
better on tests like the SAT but also in college.

So it's not an achievement measure, which would be redundant with what
grades are. But it's certainly not an IQ test which would be an innate measure
of ability. It's much more developed reasoning--the type of skills students
develop over an extended period of time.But the SAT test was basically a stepchild of the army IQ test. Right?

Right.Even when it was first being proposed, Henry Chauncey would have to
say--no, it's not measuring achievement. It's measuring ability more like an
IQ test. Did it change then from that?

The SAT in, in the past 40-50 years has changed remarkably. Just as
cognitive ability tests in general--that field has changed remarkably. The
types of items and the way we consider intelligence and aptitude and
achievement today, has evolved in the last 40 years. And tests have gotten
much better and more accurate in what they're doing.

What does the SAT measure?

The classic phrase is that these tests measure what they test. And the SAT
is no exception to that. The way items get on that test is the way items get on
most tests of mental ability which is that they are items that correlate with
performance in school. So an item that you would give to a norming sample that
doesn't correlate very well with school success gets dropped off the test.
Items that do correlate get put on the test. That's how tests get made up.
They're just empirical creations, creations of American and European
pragmatism. If you want to find out what actual mental capacity they measure
you have to work backwards. You have to use statistical techniques to classify
the kinds of performances that they're measuring and work backwards to "Well,
if it measures this cluster of performances, maybe it measures this kind of
capacity." And then there have developed big arguments about which performance
this cluster measures and what performance that cluster measures and which are
central to performance. So it's a very complicated game trying to work
backwards and figure out what these tests actually measure.But is this SAT an IQ test?

It is in a sense an IQ test. The SAT and IQ test correlate very highly.
Between the SAT and the IQ, they correlate almost as much as the SAT correlates
with a second administration of the SAT, as much as it correlates with itself.
So they're very similar tests in content. Give me the little history lesson.

The methodology for standardized tests of the kind that we use today was
developed in the 19th century by Francis Galton who was as many say, the
jealous cousin of Charles Darwin. And he was trying to get a test that would
test his kind of evolutionary, social Darwinist hypothesis that intelligence
ran in families. Of course all kinds of other things ran in families like
wealth, advantage, and so on, but that didn't bother him. He wanted a test that
would discriminate between basically upper class and lower class Brits.

He developed this technology of finding items and seeing how much they
would correlate with other performances as a criteria for whether the item
would be put on the test or not. So he had this situation in the British museum
I guess where he would have people come in and perform tasks: reaction time
tasks, visual acuity tasks, a whole variety of kind of physiologically-rooted
tasks that he thought would tap into intelligence, sort of innate, physical
intelligence. His presumption was that upper class Brits would do better on
these things than the lower class Brits and he would therefore have a set of
items that he could give to people that were a measure of intelligence that
would discriminate. People who would score high on this would be more likely to
be the upper class Brits. People who scored low on this would be the lower
class Brits. So, he died a failed scientist, never finding a set of items that
worked like that.

Alfred Binet in Paris at the turn of the century, beginning of the 20th
century, was given a practical task of coming up with a test that would help
identify kids who were retarded and wouldn't do well in school. So he simply
used Galton's technology. He said, "Well, I'll make up a bunch of items. And
the items that kids who do well in school get right, I'll put on the test. And
items that kids who don't do so well at school get right, I'll put those off
the test because they couldn't be measuring something relevant to school
success." So he gets a subset of items that kids who do well in school can
perform well on, and now he's got a test that when given to people will tend to
identify those who are not going to do well in school. And he can do what the
Paris school board asked him to do: screen out kids who are going to have real
trouble with school.

Well, as everybody knows, that became the basis of the IQ test. It was
transported into the United States, the Stanford-Binet test. That same
technology of using success in school as a criteria for whether an item gets
put on a test or taken off of a test. And that is how essentially, roughly
speaking, all standardized tests are constructed. The SAT, the GRE, the mini-IQ
test all have that inherent methodology to them. The man who developed the SAT, Carl Brigham, was an outright racist. Do
you even mull that fact?

As I say, that fact has not been wasted on me. And the area of standardized
testing and intelligence testing has always been one of the most controversial
areas of psychology for precisely that reason. It has often been used as a way
of implementing racist intent, most recently with regard to blacks. But in the
post-World War I wave of immigration, it was used to screen out Southern
Europeans, Jews, and other groups who did not score well on tests at that
particular time. So it has, as a tool, a very, very racist past.What do you think of the SAT, personally?

I think it is an exam that can tell you something. I've used a metaphor, if
you can indulge that, that I think captures the basic argument I would use. If
you had to select a basketball team by the number of 10 free throws that a
player could hit, the first thing you'd worry about is selecting a basketball
player based on how they shoot free throws and you know you'd never pick
Shaquille O'Neal because he's terrible at free throws even though he's a
magnificent basketball player. That's what a standardized test is, compared to
the domain of real school performance. Real school performance out there--it's
like having to select a basketball player based on how well they shoot free
throws. That's the first problem with standardized tests.

And the SAT reflects that. The predictive statistics reflect that. The SAT
measures only about 18%, [an] estimate range from 7 to 25%, of the things that
it takes to do well in school. This is something that people should realize
about the test. People think of it as capturing a very large proportion of
things that are important to school success. The people that make these tests
tell us, "No, that is not true. They don't capture a large portion of the
things--about 18%." In many of the samples I've done research on, much smaller
than that, sometimes 4% of the things that are predicting success in college
for example. So it's not great, just like a free throw is to selecting a
basketball team. And SAT is not going to get you very far with predicting who's
going to do well in college. And certainly not far with regards to who is going
to do well in society or contribute to society. It's just not that good a tool
and that's the first thing to realize about it.

The second set of problems have to do with interpreting the scores on SAT
tests. And again, the free throw example is useful. If a kid comes in and he
shoots 10 out of 10 or zero out of 10, you might take note of that kind of
performance with regard to selecting him on the basketball team. If he hits 10
out of 10, you say, "Well, okay, he's probably pretty good and that probably
reflects something about his basketball playing. I'll put him on the team. Zero
out of 10, that probably reflects something about his playing, he's off the
team." Same with SAT tests I think. When you get really strong scores one way
or the other, even though they're not as reliable, they often can bring to
light talent that would not otherwise be seen.

And so I am not one who thinks they should be done away with entirely. They
can be useful in that regard as long as we understand how to interpret them and
how little to use them. And I think many college admissions committees are very
sophisticated about this. They are closer to this issue of how predictive tests
are, and they can get a feel for it. So, that's the second thing.

Middling scores on the test are very difficult to interpret because you
don't know. If the kid practiced a little bit more, maybe he would have hit 9
free throws. Maybe he hit only 4 and he's been practicing for 10 years. It's
just hard to interpret the meaning of middling scores and the same is true with
the SAT. A kid who gets anywhere from 10 to 1200, maybe he got those scores
because of coaching or maybe he got those scores because he didn't have enough
coaching or maybe he got those scores because he went to Europe every summer
and got a great vocabulary about cathedrals and that happened to be on the test
that day. All kinds of things can contribute to performance and it muddies up
the diagnosticity of the test.

Do we or don't we have a neutral and impersonal meritocracy measuring
merit?

Well, it is certainly impersonal. I don't know I'd go so far as to say
it's either neutral or meritocratic. FairTest, for example, would say
that these tests--and I want to be clear, I'm [not] talking about all tests.
I'm a professor, I believe in methods of evaluation. I think some methods are
not only more fair but also more valuable. And what I'm talking about here in
the guise of tests is aptitude testing, tests which are used to predict future
performance, not tests which are used to give feedback, either to the teacher
to the student, as to what they have actually mastered or what they are
learning. I'm not talking about diagnostic tests. I am talking only about
aptitude tests. Because it is the aptitude test that we are using as the proxy
for merit. And it is as if this test functions as a thermometer. And you give
each person the test as if you were taking their smartness temperature. And
that unfortunately, is not how the test functions. Even the test makers do not
claim it is a thermometer of smartness. All they claim is that it correlates
with first year college grades. And if it's the LSAT, with first-year law
school grades.

Now, correlate--that's a big word. What does correlate mean? There's some
consistency. There's some relationship between the score on this aptitude test
and your first year college grades. That's true. There is some relationship.
The problem is it's a very modest relationship. It is a positive relationship,
meaning it is more than zero. But it is not what most people would assume when
they hear the term correlation. For example, your height correlates better
with your weight than your test score correlates with your first year grades.

Jane Balin, Michelle Fine and I did a study at the University of
Pennsylvania Law School where we actually looked at the first year law school
grades of 981 students and then looked at their LSAT scores. And it turned out
that there was a relationship between their LSAT and their first-year law
school grades. The LSAT predicted 14 percent of the variance between the first
year grades. And it did a little better second year: 15 percent. And I was at
a meeting with a person who at the time worked for the law school admissions
council who constructs the LSAT. And she said, well, nationwide the test is
nine percent better than random. Nine percent better than random. That's what
we're talking about.

So it may be an efficient tool in that you get the students to pay for it.
The schools don't pay for it. It allows the schools to then rank order people
based on a number that is assigned to them. But it is a fairly arbitrary tool.
And it is certainly not a thermometer of merit, if by merit--and I'm assuming
we don't mean merit is the equal of first-year college grades or first-year law
school grades. Merit is a big word. And it has to carry a lot of weight. It
does a lot of heavy lifting. It means more than just how you're going to do
first year in college. Because if all we cared about is how well you do first
year in college, we would have college as one year. Right? Why would you have
to be there and pay tuition for three more years if this is only about first
year of college? If it's such a good predictor, why do you even go to college?
Just take the test and then get a diploma.

So there must be something going on within the institution of higher
education or within the legal academy that we think also carries, quote, merit.
In which people are learning how to work and play well together with others, in
which people are learning intellectual self-confidence, in which people are
being exposed to research skills, in which people are being trained to be
leaders. None of this has any relationship to the testocracy. No one claims
that aptitude tests predict leadership, predict emotional intelligence, predict
the capacity to make a contribution to the society. The only relationship is
between the test and first-year college grades.

And what I was about to say earlier was that, with FairTest and others,
they will say that what the test actually judges is quick strategic guessing
with less than perfect information. Boys, for example, do better on the math
portion of the SAT than girls. They routinely score 40 to 50 points higher.
Many people say, well that's because girls are ignored in high school math.
That may be true. And yet the girls do just as well in college when they take
math courses as the boys, despite their lower SAT scores on the math portion.
And when you interview the boys as to how they approach the test, the answer is
they basically viewed it as a pinball machine. And the goal was speed and
winning. And the girls on the other hand, wanted to work through the problems
before they put down the answer. That, apparently, is not merit.

Somebody who wants to work through a problem before concluding with an
answer, is not guessing and they're not fast. And so on some level, what we
are confusing as a result of this over-emphasis on the testocracy--what we're
confusing merit with is speed and the confidence to guess.

Isn't one of the criticisms of the SAT that nobody's quite sure of what
it does measure? Is it because ETS can't say, or don't want to say, what it
measures?

I think it's hard to say exactly what it measures. And I'm very
sympathetic. It's especially hard to say what the SAT measures if you want to
keep the acronym SAT, which is most successful marketing tool in testing
history. Well IQ is right up there too, but SAT is what everybody knows you
have to take to go to college and it's the test that's marketed by ETS. So if
they change the name in any way that does reproduce SAT, they're in real
trouble. So I think they've got a constraint there.

But in fact, if from the beginning it had been called, say the Scholastic
Achievement Test, I think it would've taken the political curse off the thing.
It isn't exactly an achievement test, but it's certainly not exactly an
aptitude test. But if you recognize in the label that this involves
achievement, then people will say, okay, well it may not be the kind of
achievement we should test, but it's reasonable that you should give a test if
it measures achievement.

Whenever Henry Chauncey, who was working with James Conant, ever got close
to the word achievement, Conant would say that's not what I want, because
achievement was then the privilege of the guys who went at that time to Exeter
and Andover.

Well I think, when these tests were originally developed, people really
believed if they did the job right, they would be able to measure this sort of
underlying, biological potential. And they often called it aptitude, sometimes
they called it genes, sometimes they called it intelligence. But whatever they
called it, they though that there was something there and if they just tweaked
and fiddled and worked at it a little harder, they would get pretty close to
being able to measure it.

I don't think people believe that anymore. They believe that how you do on
almost any test is substantially affected by both your heredity and your
environment and both things make a difference. Any psychologist would tell you
that. But, the problem of finding a label for something which is both A and B,
is a tough one and you could say it's the Scholastic Aptitude and Achievement
Test for instance, but that doesn't sound good when you're selling it.
Especially to someone like Conant who wanted a test that measured aptitude.

I mean it wasn't an accident that they came up with these terms. They came
up with the terms because that's what people wanted to do. The fact that they
didn't actually quite do it was, you know, well we've seen this in a lot of
other fields of merchandising too. You know, people want a product that does
so and so, then say it does so and so.What's the only way to eliminate labeling bias in tests?

I think you've got to re-label these tests. You need to call these tests
things that really reflect what it is that they measure. And then it's also a
question of whether what kinds of things the tests should really measure that
you give, but that's a different question.Do the SAT's do a good job of predicting academic
performance?

I think they really don't do a good job of predicting. They do a pretty
poor job of predicting, but they're the best we've got. Well that means the
people who are good at those things, high school grades and test-taking, are
going to do well in getting into good colleges. And other people who would do
equally well in college are just not going to make it because we have no way of
picking out the kid who will do well even though his high grades weren't so
great and even though his SAT scores weren't so great.

There are lots of kids out there like that and on the average they won't do
as well as the other ones and the ones who will, are going to lose out because
we can't identify. They come with little tags. If we knew what else it was,
you know, if we could say, well it's stick-to-itiveness or, it's getting
excited by a teacher or something, then we could measure it. That would help a
lot. And it would probably help minority kids in particular because they don't
do well on these tests and they are put at a disadvantage by that, and that's
even more of an issue on the job where we know that tests are not terribly
strong predictors of job performance and we know that lots of other stuff
counts, But we don't know how to measure most of the stuff except by hiring
somebody and seeing how they do.Do you think we're misguided in using them?

I think we'd be way better off if we gave achievement tests and didn't
emphasize the so-called aptitude test or now, just the mysteriously unlabeled
SAT. I don't think it would change the results in favor of minorities to any
great extent in the short run, but I do think it would have a good effect in
the long run.

And the reason it would have a good effect is that if you start testing
achievement you send a measure to people that this is what you've got to learn
to go to a good college, or to any college, whatever. And we know from all
kinds of evidence that if you actually set a task like that, the minority
students can do better than they're now doing. So I think that if we kind of
change the way we set up the task and said this is a question of achievement,
it's just like lots of other forms of achievement. You've got to work hard it,
you've got to practice, you've got to get good at it.

You would have a very different state of mind than when it seems to people
that this is something that is aptitude, unchangeable, inborn, you know, if I
can't do, I just can't do it. That's a signal for defeat and giving up. And
of course, it's not just a signal to minorities for giving up, it's a signal to
any kid who tests badly and says, gee, I just don't get good scores on these
kinds of tests. Whereas if you tell him, you know this is a math test, you have
to understand the test, lots of people can learn that math if they work hard at
it.