Kindly Log In

Debunking mediocre performance of U.S. students

January 15, 2013

January 15, 2013

The conventional wisdom that American students lag far behind top performers like Finland and South Korea in academic achievement is oversimplified. A new study out today by researchers at Stanford University and the Economic Policy Institute finds that comparisons of scores on international tests fail to adequately consider social and economic differences.

PISA scores for U.S. students increase when adjusted for social class. Source: What do international tests really show about U.S. student performance? (Click to enlarge).

“If the social class distribution of the United States were similar to that of top-scoring countries, the average test score gap between the United States and these top-scoring countries would be cut in half in reading and by one-third in mathematics,” write Stanford Education Professor Martin Carnoy and Richard Rothstein, a research associate at the Economic Policy Institute, in their report, What do International Tests Really Show About U.S. Student Performance?

The study also found big discrepancies in results for different tests. On the 2009 Program for International Student Assessment (PISA), U.S. students scored 487 in mathematics while Finnish students scored 541. Two years later, on Trends in International Mathematics and Science Study (TIMSS), U.S. students scored 509, just five points below Finland.

Carnoy and Rothstein also say it’s not true that low-income American students do significantly worse on the tests relative to their high-income classmates, while the economic achievement gap in other countries is much smaller. The U.S. has the smallest gap on reading and math on the PISA exam compared to France, Germany and England. And next to the top-scoring countries, the U.S. has a narrower achievement gap than Korea, and isn’t far behind Canada and Finland.

However, the authors do acknowledge that U.S. students score lower than students in the top-ranked countries across the board at every economic level. “At all points in the social class distribution, U.S. students perform worse, and in many cases substantially worse, than students in a group of top-scoring countries (Canada, Finland and Korea). Although controlling for social class distribution would narrow the difference in average scores between these countries and the United States, it would not eliminate it,” they write in the study.

On the 2009 PISA exam, which assesses 15-year-olds, the united States ranks 14th in reading and 17th in science, which are about average, but drops below average in mathematics to 25th. On TIMSS, which tests students in fourth, eighth and twelfth grades, U.S. students were above the international average in both science and mathematics in grade four, above the international average in science and below the international average in mathematics in eighth grade and among the lowest in both science and mathematics for twelfth grade students.

The message for policymakers is to take a more nuanced view of test scores, not to rely on just one exam, and not to jump to the conclusion that U.S. students are unprepared to compete in the global economy, said the researchers.

“Such conclusions are oversimplified, frequently exaggerated and misleading,” said Rothstein in a news release on the study.

Not so, says Andreas Schleicher, Deputy Director for Education and Special Advisor on Education Policy at the Organization for Economic Co-operation and

Scores on PISA adjusted by the number of books children report having in their homes. Source: What do international tests really show about U.S. student performance? (Click to enlarge).

Development, the group that administers PISA. He said the report “contains several fundamental misunderstandings and misinterpretations of the PISA data. In particular, the paper claims that there are flaws in PISA samples, which is simply incorrect and unsupported in the paper.”

The primary factor distinguishing students by social class, according Rothstein and Carnoy, is how many books a child has in his or her home. “Books in the home, to us, refers to the academic orientation of the household,” said Carnoy. He and Rothstein also say that this “indicator of household literacy” is used by both PISA and TIMSS and is “plausibly relevant to student academic performance.”

Going deeper

What do International Tests Really Show About U.S. Student Performance?

Comments Policy

Gary Ravani7 years ago7 years ago

“Books in the home” correlates to family wealth. And family wealth is still the most powerful predictor of academic achievement.

See research done by Sean Reardon @ Stanford.

CarolineSF7 years ago7 years ago

Aside from the question of the validity of the data, the Books In Home concept seems to raise the usual issue of correlation vs. causation.

edfundwonk7 years ago7 years ago

I’ve heard the old saw about the number of books in the home being the best predictor of academic achievement many times. I have several problems with this statement.
First, it implies that the type of book or its level of difficulty is irrelevant – that a library of 100 romance novels is as good a predictor of academic success as a library filled with 100 classic works of fiction and nonfiction. Second, the researchers … Read More

I’ve heard the old saw about the number of books in the home being the best predictor of academic achievement many times. I have several problems with this statement.

First, it implies that the type of book or its level of difficulty is irrelevant – that a library of 100 romance novels is as good a predictor of academic success as a library filled with 100 classic works of fiction and nonfiction. Second, the researchers offer no proof to back up their claim, simply stating that the number of books in the household is “plausibly relevant to student academic performance.” Finally, even if it is true that “the more books, the higher the academic achievement,” the finding has precious little to offer in terms of effective public policy. I doubt whether increasing the number of books in every household would have any impact on student achievement.

el7 years ago7 years ago

It's a good question, about Books In Home, and obviously if you respond to it by simply dumping off a truckload of remaindered books at every home in America, you've missed the point.
But, absent that, what it does say is "this household values books and reading." I suspect it matters less than you think what the content of the books is. This household keeps books, displays them, has them available.
Of course, in the new era … Read More

It’s a good question, about Books In Home, and obviously if you respond to it by simply dumping off a truckload of remaindered books at every home in America, you’ve missed the point.

But, absent that, what it does say is “this household values books and reading.” I suspect it matters less than you think what the content of the books is. This household keeps books, displays them, has them available.

Of course, in the new era of digital readers, its relevancy may also change quite a bit in the next decade or two, when kids who grew up reading on tablets start setting up their own households and starting families.

Joan Brownstein7 years ago7 years ago

The cure is not to parse the tests after students are in school. The cure is universal pre-school, for the period of greatest brain growth is in the first three years. If children come to kindergarten without the skills to succeed, the road to scholastic success will be long and hard, and not a sure bet.

Educator7 years ago7 years ago

Professor Yong Zhao also questions the usefulness of the exams themselves. Here are some thoughts to ponder:
1) Do we even know that high scores on these multiple choice tests leads to a better society?
2) If the U.S. has stunk on these international exams for so many decades, why is the U.S. still around and still a leader?
I suppose one could argue there does need to be some sort of objective way to measure learning … Read More

Professor Yong Zhao also questions the usefulness of the exams themselves. Here are some thoughts to ponder:

1) Do we even know that high scores on these multiple choice tests leads to a better society?

2) If the U.S. has stunk on these international exams for so many decades, why is the U.S. still around and still a leader?

I suppose one could argue there does need to be some sort of objective way to measure learning internationally. I just worry when these exams are used to push education reforms by folks who have little to no education experience, and so much emphasis is placed on it.

CarolineSF7 years ago7 years ago

This is a really important point from @el, and EdSource might really enlighten the public if you made an effort to spread this information:
"Something to keep in mind is that America has never scored at the top of these tests, and yet as a whole our nation has been very successful with science and technology. The slacker kids who did poorly on those tests in the 70′s are the core of our workforce today."
It's almost … Read More

This is a really important point from @el, and EdSource might really enlighten the public if you made an effort to spread this information:

“Something to keep in mind is that America has never scored at the top of these tests, and yet as a whole our nation has been very successful with science and technology. The slacker kids who did poorly on those tests in the 70′s are the core of our workforce today.”

It’s almost universally believed, as God’s unquestioned wisdom, that U.S. students used to be No. 1 and have slid. If you look around you, you can see this unquestioned assumption stated as fact (unattributed) very often. I just saw it in a wire story about the issue of year-round school vs. summer vacation. Obama said it in a speech when he was first elected.

I heard Prof. Yong Zhao of the University of Oregon do a presentation on this a couple of years ago (he’s really amusing, too); and I researched it for a project I did for a class last year. Actually, the U.S. has been in the middle of the pack for the entire time these international comparisons have existed — since the early-mid ’60s. We were in the middle of the pack when we won the space race, when we led the world in innovation and creativity, and when we were the No. 1 powerhouse economy in the world. It’s just plain false that the U.S. was ever No. 1 — or even close to the top — in international test score comparisons, and it’s false that we’ve fallen.

This inaccurate belief is at the heart of a lot of misguided education “reform” ideas, and it may steer well-intentioned reformers wrong.

el7 years ago7 years ago

There are many excellent nuggets in the full report. I hope people interested in the topic of international comparisons will take the time to read through it in full.
Here's another quote from it:
------------
"Other considerations, rarely considered in public debate, also influence the care we should take in the interpretation of international comparisons. One is how the curriculum is sampled in the framework for any particular test. Because the full range of knowledge and skills that … Read More

There are many excellent nuggets in the full report. I hope people interested in the topic of international comparisons will take the time to read through it in full.

Here’s another quote from it:

————

“Other considerations, rarely considered in public debate, also influence the care we should take in the interpretation of international comparisons. One is how the curriculum is sampled in the framework for any particular test. Because the full range of knowledge and skills that we describe as “mathematics” cannot possibly be covered in a single brief test, policymakers should also carefully examine whether an assessment called a “mathematics” test necessarily covers knowledge and skills similar to those covered by other assessments also called “mathematics” tests, and whether performance on these different assessments can reasonably be compared. For example, American adolescents perform relatively well on algebra questions, and relatively poorly on geometry questions, compared to adolescents in other countries. Reports on how the United States compares to other countries show the United States in a more favorable light to the extent a test has more algebra items and fewer geometry items. Whether there is an appropriate balance between these topics on any particular international assessment is rarely considered by policymakers who draw conclusions about the relative performance of U.S. students from that assessment. Similar questions arise with regard to a “reading” test.

“Whether U.S. policymakers want to reorient the curriculum to place more emphasis on geometry is a decision they should make without regard to whether such reorientation might influence comparative scores on an international test. It certainly might not be good public policy to reduce curricular emphasis on statistics and probability, skills essential to an educated citizenry in a democracy, in order to make more time available for geometry. There are undoubtedly other sub-skills covered by international reading and math tests on which some countries are relatively stronger and others are relatively weaker. Investigation of these differences should be undertaken before drawing policy conclusions from international test scores.”

Emphasis, if the formatting came through, is mine.

el7 years ago7 years ago

Finally, the emphasis on rank in these international comparisons is misleading as well. It would make more sense – if you cared about the actual data rather than having a nice scary boogeyman – to discuss deviations from the mean and deviations from the top. In some instances, you’ll see several countries clumped together within the margin of error, making rank an inappropriate measure of Educational Goodness even within the limitations of the testing scheme.

el7 years ago7 years ago

"In December 2012, the International Association for the Evaluation of Educational Achievement (IEA) released national average results from the 2011 administration of the Trends in International Mathematics and Science Study (TIMSS). U.S. Secretary of Education Arne Duncan promptly issued a press release calling the results “unacceptable,” saying that they “underscore the urgency of accelerating achievement in secondary school and the need to close large and persistent achievement gaps,” and calling particular attention to the fact … Read More

“In December 2012, the International Association for the Evaluation of Educational Achievement (IEA) released national average results from the 2011 administration of the Trends in International Mathematics and Science Study (TIMSS). U.S. Secretary of Education Arne Duncan promptly issued a press release calling the results “unacceptable,” saying that they “underscore the urgency of accelerating achievement in secondary school and the need to close large and persistent achievement gaps,” and calling particular attention to the fact that the 8th-grade scores in mathematics for U.S. students failed to improve since the previous administration of the TIMSS.”

I guess this is a clear indication that using Value-Added measures, that Arne Duncan has been unsatisfactory as Secretary of Education.

——-

Something to keep in mind is that America has never scored at the top of these tests, and yet as a whole our nation has been very successful with science and technology. The slacker kids who did poorly on those tests in the 70’s are the core of our workforce today. So, I think it’s worth asking if those tests really are valid indicators of our future and the quality of the workforce we produce. I don’t know the answer, but I don’t think we should assume that they are.

And finally, the fact that 25% of American kids live in poverty, and that that poverty line is set incredibly and unlivably low, is perhaps the most shameful and alarming statistic of all. I suspect we will do more to improve education outcomes with universal health care than we will with all the various reformy proposals going around right now.

navigio7 years ago7 years ago

From the executive summary: "There is no precise way to make social class comparisons between countries. PISA collects data on many characteristics that are arguably related to social class status, and also assembles them into an overall index. Although none of the possible indicators of social class differences is entirely satisfactory, we think one, the number of books in the home (BH), is probably superior for purposes of international test score comparisons, and we use … Read More

From the executive summary: “There is no precise way to make social class comparisons between countries. PISA collects data on many characteristics that are arguably related to social class status, and also assembles them into an overall index. Although none of the possible indicators of social class differences is entirely satisfactory, we think one, the number of books in the home (BH), is probably superior for purposes of international test score comparisons, and we use it for our analysis. A very high fraction of students in both the PISA and TIMSS surveys answer the BH question, something less true for other important social class indicator questions asked on the student questionnaires. As we explain in greater detail below, we also examine whether other social class indicators, such as mother’s education or PISA’s overall index, in addition to BH, would produce meaningfully different results, and determine that they would not. We conclude that BH serves as a reasonable representation of social class (home) influences on students’ academic performance.”