Hysteria over PISA misses the point

By
Valerie Strauss

Finland is so over. Now it’s all about Shanghai.

The 2009 results released today from the Program for International Student Assessment, known as PISA, caused consternation in the United States today when American students racked up generally average scores in reading, science and math. Where they’ve been for years.

Today’s big news: Students from Shanghai, participating for the first time in the program, came out on top in all three areas out of about 65 countries and other education systems.

Here come the Chinese, or, rather, the Shanghainese.

You’d think it would be the Finns who would be beside themselves: They lost their top literacy ranking to South Korea; the United States was 17th. In math, Sinagpore was second in math; the United States, 31st. In science, Finland, was in second place; the United States, 23rd.

U.S. Rep. George Miller (D-Calif.), chairman of the House Education and Labor Committee: “ .... Average won’t help us regain our global role as a leader in education. Average won’t help our students get the jobs of tomorrow. Average is the status quo and it’s failing our country. This is clearly an issue we need to tackle in the next Congress ... ”

And then there was Chester E. Finn Jr., a former assistant secretary of education and president of the Thomas B. Fordham Institute, who wrote on his Flypaper blog:

“On Pearl Harbor Day 2010, the United States (and much of the rest of the world) was attacked by China.

“Too melodramatic? Maybe you’d prefer 'Sixty-three years after Sputnik caused an earthquake in American education by giving us reason to believe that the Soviet Union had surpassed us, China delivered the aftershock.'

“It came via yet another wonky study, The PISA 2009 Results: What Students Know and Can Do, reporting that on a test of math, reading and science given to fifteen year olds in sixty-five countries in 2009, Shanghai’s 15-year-olds topped those in every other jurisdiction in ALL THREE SUBJECTS. What’s more, Hong Kong ranked in the top four on all three assessments ....

“... Will this be the wake-up call that America needs to get serious about educational achievement? Will it be the Sputnik of our time? Will it stir us out of our torpor and get us beyond our excuse-making, our bickering over who should do what, our prioritizing of adult interests and our hang-ups about the very kinds of changes that China is now making while we dither?"

Well, maybe.

But let’s not get ahead of ourselves here.

Shanghai is not representative of the entire Chinese population, and China makes no pretense of trying to educate the entire populace, as we do. Hong Kong, of course, is not ruled by Beijing.

But it is a test-driven society (an educational culture that Finn himself did not find when he attended Exeter Academy), which actually is right in line with today’s American school reform philosophy.

But our high-stakes standardized test obsession, the ones mandated by No Child Left Behind, have, apparently, done nothing to improve the reading, science and math literacy of American 15-year-olds … if, that is, you put a lot of stock in the results of one international testing system. And even if you don't.

For nearly a decade, public schools have been test-obsessed, and charter schools have abounded. Those who hold test scores as important measures of progress should face the obvious: NCLB didn't work.

And that is something Congress should seriously consider when it decides whether, and how, to change No Child Left Behind.

Some details from the PISA report on the performance of U.S. students:

• U.S. 15-year-olds had an average score of 500 on the combined reading literacy scale, not measurably different from the OECD average score of 493. Among the 33 other OECD countries, 6 countries had higher average scores than the United States, 13 had lower average scores, and 14 had average scores not measurably different from the U.S. average. Among the 64 other OECD countries, non-OECD countries and other education systems, 9 had higher average scores than the United States, 39 had lower average scores, and 16 had average scores not measurably different from the U.S. average.
• U.S. 15-year-olds had an average score of 487 on the mathematics literacy scale, which was lower than the OECD average score of 496. Among the 33 other OECD countries, 17 countries had higher average scores than the United States, 5 had lower average scores, and 11 had average scores not measurably different from the U.S. average. Among the 64 other OECD countries, non-OECD countries, and other education systems, 23 had higher average scores than the United States, 29 had lower average scores, and 12 had average scores not measurably different from the U.S. average score.
• On the science literacy scale, the average score of U.S. students (502) was not measurably different from the OECD average (501). Among the 33 other OECD countries, 12 had higher average scores than the United States, 9 had lower average scores, and 12 had average scores that were not measurably different. Among the 64 other OECD countries, non-OECD countries, and other education systems, 18 had higher average scores, 33 had lower average scores, and 13 had average scores that were not measurably different from the U.S. average score.

There are places in the world where teachers are highly respected and students' individual rights are secondary to the rights of the group. I suspect that the kids in Shanghai work hard in school and do not think disrespecting teachers is cool.

Hysteria over PISA misses the point.
Agreed. The point is that the U.S. kids are once again average (or really a bit below) and nothing is likely to happen. Sputnik, no way. That was about motivating our top scientists. Infinitely harder to raise the whole ship unless we are willing to admit that we don't have the answers and ask China for help. NCLB passed both Houses of Congress with 90% support. It was our best shot. China clearly has some answers and we don't. Here are some suggestions I made previously.
Emulate China.
Adopt more Confucian programs in our elementary schools.
Send hundreds of teachers to Shanghai to study what they are doing. Arrange thousands of student exchanges with Chinese schools so that our students can get a taste of what it is like to really work.
China not only created an economic miracle in the past 30 years; they are in the process of creating an educational one as well. Wake up and smell the tea America.

Self-disclosure- I was a teacher for many years in U.S. public and parochial high schools. I now live in Beijing and write a regular weekly column for China Daily Online.

Let's settle down and stop making policy by panicking again. Waiting for Superman? Panic! The Lottery? Panic! NCLB? Panic! Has anyone spent any time trying to compare who is taking the PISA test in Shanghai versus US test takers? Most of the top jobs in the US both political and commercial are filled by kids from the most elite high schools and universities. If the cohort in this testing was drawn from that group I would guess that the US would do much better. Also, the Chinese have a way to compel serious taking of these tests the US has no such leverage. Are 15 year olds motivated to due well for the US? There are no consequences for the individual students. Have we ever checked the SAT/ACT scores by country? Enough with the PISA already. We can't even accurately compare scores across states let alone between countries with materially different cultures, ethnic diversity and economic diversity. Can anyone provide some comments on my gut feelings here?

The writer is still in denial. The only solace we have is that we have stronger and more innovative academic institutions. But when I look around my lab and labs of my colleagues (in one of the major US research universities), they are populated by Chinese, Indian and Russian post-docs and graduate students. When I posted a position last year in Science Journal, 90% of the respondents are from oversea, and no American responded to the advertisement. I ended up hiring a Chinese national living in the US. The adequacy of Math and Science education in high school will determine whether an individual pursues science and technology career later. We have been relying on foreigners all these years, it has reached crisis stage.

The problem is the sample of students tested in Shanghai is *no way* representative of the country. It's like taking the test scores from the students at Stuyvesant High School in New York as representing the whole city. According to this page about 10% of the Chinese population have only a primary school education, about 15% have completed high school, and about 7% have finished college.

Showing the results for Shanghai paints a very misleading picture of the state of education in China.

I'll say one positive thing for NCLB and the present "reform" movement; it HAS forced us to think about education and especially about all the American children who haven't had an equal opportunity to obtain one. We've learned much by studying successful students in this country and in others:

Education is a highly complex process, requiring a partnership among schools, parents, students and communities. All participants are very important so if you take out one, the child's educational opportunities could be seriously compromised.

The teacher is important. All countries with successful systems of education value their teachers, treat them well, and have extremely high standards for entry into the profession.

The most crucial years in anyone's education could very well be the first five years of life when cognition and language develop rapidly.

I have listed some factors that we know for certain are important for the education of children. We could start heeding them today and see positive changes within a few years. Following clueless leaders as we're doing now is just asking for more disappointment in the coming years. And of course NCLB should be scrapped, unless we want to see more of the same.

Wow, excuse me!?! "Liberals should be happy"???? You may be interested to know that for the past 25 years schools have been beaten to death by a "back to basics" movement that has consistently narrowed school options until American K-12 education has all but reached the vanishing point on the global scene. The backwardness of our national enthrallment with a steady and increasing diet of "standardized testing" to the exclusion of all creative, critical, or even sensible thinking is directly to blame for the end-game in which we are currently engaged.

Our system is obsolete and nothing is more obsolete than the "conservative" belief that the corporate culture based on numbers, data and the bottom line can somehow magically transform schools. That decades-long narrowing effect has done nothing but egregiously hold the line dividing haves and have-nots, efficiently delineating who shall succeed and who shall not in what we like to think of as our free, democratic society.

The difference for the U.S. is that, unlike China, which is just now coming into its own as an industrial nation, we've been there, done that. Our next step is to capitalize on our knowledge about how the human brain really works (a little like a super-charged worldwide web) and REINVENT our public school system to take advantage of that biological fact. Americans (of all ethnicities and origins) really believe in our national ideals: progress, independence, self-determination, etc. and we actually care about individual people. If we really want to move forward as a nation, then we need to be talking about how to emulate and demonstrate those characteristics in our schools: personalized, humane, supportive, inspiring. We're wasting tons of human capital messing around with the symptoms of decay (drop-outs, drugs, truancy, discipline, suicides even) when we need to be looking at the CORE BELIEFS that we share (and that are not business-oriented at all).

New vision for schools, anyone? We can only move back into a global leadership position when we realize that all children have inestimable gifts and design public education to help teachers and parents bring those seeds of greatness to life. That's the best investment we can make in both a better world and our own economic future. Check out a brand-new campaign to build that new vision at www.facebook.com/changetheschools and speak up for moving FORWARD!

I'm with Valerie on this one to a point. We are getting our shorts twisted over a snap shot without looking at the background. We are reacting to American scores against a specific society, hand picked by the Chinese. I'm not upset at all.

Faced with other nations, I still want to know that our children are on a level playing field. What are the education standards and mandates by comparison? Were they, as the Chinese, hand selected in a particular section of their society?

If the hours, days, programs, are not the same, then let's analyze how other countries got there. We can improve and we should, but before we make kids feel like they are inferior let's get the landscape comparable.

Cryos--???????
Are you from a backwards, inside out dimension? Since 2002 and the advent of NCLB and Bush's administration, conservative business interests have been calling the shots as they decide education policy and direct tax dollars into the pockets of privately owned companies that produce redundant test materials. It's all about the money.

And if the curriculum includes too much softball stuff, it isn't because the teachers mandated it. If you don't like it, go complain to your school board. Teachers get in trouble when they complain about garbage curriculum.

People please look at the data. This was not a hand-picked sample from Shanghai. It was a random sample of students from a city with 20 million people (about the size of NYC and LA and 5 times as big as Finland, the perennial PISA leader).
Does China educate all her students equally? No. There is still a long way to go before the education of migrant children catches up. Rural education is still far behind. But look at some of the quotes from the American educators who actually oversaw this study. Here's one from Chester Finn in today's WSJ.
"If China can produce top PISA scorers in one city in 2009—Shanghai's population of 20 million is larger than that of many whole countries—it can do this in 10 cities in 2019 and 50 in 2029. Or maybe faster." China is doing for large populations of students on less money what the U.S. hasn't been able to do.
And not just in science and math. The country is beating our pants off in reading too.
We don't need to adopt China's model wholesale and besides that would be a practical and political impossibility. What we should do is ask them to help us get our house in order.

Thankfully the Washington Post will help reverse this trend...by promoting Sarah Palin (a person who couldn't tell you what an atom is, or what water is, or why it rains), her anti-education, anti-knowledge agenda along with her daughter Bristol, an unwed teenage mother and illiterate, unemployed* high school drop-out as the model for American youth.

We don't need the Chinese to tell us how to get our house in order. We don't need to throw more money at the problem. KIPP schools work without tons of federal tax dollars thrown at them - they require parental concern and dedication from everybody involved. It's not calculus. It's willpower.

You are absolutely right that "Shanghai is not representative of the entire Chinese population". Actually, in the old days (before late 1980s) when the whole China had a unified version of Gao Kao (the national college entrance exam) papers, Shanghai students' average score is routinely among the lowest echelon of the league. In order to cover up the obvious unfair seat allocation scheme favoring Shanghai, the city was later allowed to produce its own version of the Gao Kao papers so the scores are no longer comparable. But still, it's obvious that Shanghai's pre-college education lags far behind many other much less developed provinces, most notably Hunan, Hubei, Sichuan (with population of 100 million), Jiangsu, Zhejiang, and Shandong. It's a common trick for the parents who have the means to move their household registration to Shanghai in their kids' exam year to afford them an unfair competitive edge in Gao Kao.

I don't think you get the point. Being rich does NOT afford better education. Spend more does NOT get you better education. I'm from the richest, most developed city in my province. But the best students in this province are from one of the poorest, rural prefecture, not the city. Rich parents from my city who want to improve their kids' academic performance would pay high fees to send their kids to those poor rural schools, let the kids mingle with the poor kids who studied so hard in order to climb the social ladder.

It's the willingness to work hard, to compete, to change their fate through education that powers China's education.

"Shanghai is not representative of the entire Chinese population"
Yes, but do you know that Shanghai's population is over 3 times the population of entire Finland or Singapore? What can be done by this many people in Shanghai alone? Go figure.

Listen to Patrick Mattimore--sounds like he taught here and lives there. These results don't seem off the wall to him. Patrick, would love to talk to you at govt.bureaucrat@gmail.com

PISA controls selection very carefully. Randomly picks schools and then randomly picks students within the schools to test.

Shanghai is not representative of the whole country but is probably a reasonable proxy for the schools in their big east coast urban centers--Beijing, Nanjing, Suzhou, Guangzhou, etc. There are enough students in these east coast schools who are likely getting a world class education to take every job in America. (Not that that is going to happen.)

At the same time they have 400+ million people who live on $1-2 per day. This is the source of enormous social stress and mass migrations. It is China's problem and they are working on it. That said, will it make you feel any better when you lose your job to someone in the paragraph above?

@fsg2118- Careful of your gut feelings.These Chinese students were probably more motivated to do well for their country than most students in most countries. But this accounts for their fearsome "atheist work ethic" too. These kids were NOT handpicked, but randomly sampled. We CAN measure things that matter. PISA scores at age 15 are correlating well with success in college. A nation's PISA scores correlate with the rate of growth of its economy. Poorly educated workers hurt!

We can definitely learn from China and other countries--a lot of what they are doing they learned from us, and then improved on it.

I do think we need to look at NCLB or at least how educators have responded. Terror, teach test, not be straight with parents about the work they will need to do. The reality is that Chinese parents will invest more in their kid by making them work harder. Many Asian parents do, they think my kid is smart all is well. But any of you that think oh this is a few folks that are handpicked in Shanghei so we can ignore this situation live in a fantasy land. Take a look at the headlines any day of the week and you will see major U.S. Businesses moving facilities and products lines there because that is where the people and money is now. 20% of U.S. takers could not reach level two for either math or reading. We are failing to educate our students. Parents, Teachers, Principals all are at fault.

1. Article says Shanghai is a magnet for the best students in China. Well you cannot just move to Shanghai and go to school there. China uses the Hukou system (residence permit). A student below college-level will most likely attend school in a city where he has Hukou, and that's almost always the city he is born in.

2. Article assumes Shanghai has the best and brightest students, but if you look at China's college entrance exam scores (Gaokao) over the past years, Shanghai is not near the top in terms of averages.

For people who have experiences with both the American and Chinese education system, the results are not a surprise.

@dumpforjunk
Not a total surprise, no. I expected Shanghai to score about where Hong Kong did. Shanghai beat Hong Kong in every category. Finished first in the world in every category and on every subscale score!
In math, they were a full standard deviation above the OECD average and almost a half standard deviation above the next nearest nation.

It did not surprise me that they did well. The magnitude of how well they did was a surprise.

xzwlists said the same: Shanghai, Beijing's avg. test scores have historically lagged behind other top-tier China provinces like Sichuan, Jiangsu and Hunan, just to name a few. These PISA exam scores will only go up if administered in these other regions of China.

The biggest misconception NYT article portrayed was Shanghai is the crown jewel of China's education system. It is not.

Several years ago I was at a wedding and noticed that there were a high number of very successful people at the reception. Many of these people had gone to schools like Harvard, MIT and Caltech and had become physicians, engineers and academics. But others were in business or the arts. Most were the great-grandchildren of Italian immigrants.

I decided to ask a lot of these people (mainly cousins of mine) what had motivated them to do well in school.Most of them gave answers that centered around "qualitative and non-tested subjects." Some of the responses were:

I found joy in reading from an early age;

My third grade teacher had us pretend to be on the Lewis and Clark expedition. I was hooked after that;

I loved school from the beginning;

I felt successful at school;

My mom bought me this great construction toy when I was little. After that I wanted to build;

Etc. Almost no one spoke of drills or test scores. One engineer did comment "that of course I knew the basics from an early age."

I also second Emily's suggestion to read the articles written by Professor Zhoa of Michigan State. We might denigrate our system of education, but the Chinese people learn from it.

fsg2118 makes sense - our overtested students do not have any motivation to perform on this test. Additionally, the PISA sampling is abominable. I used to direct the research office at an elite high school, and they were begging me to use our totally unrepresentative students (I declined). The international comparisons currently are junk science.

This does not mean that we don't have a problem. The attention spans of our students are eroding to gnat level, and it shows in their abstract thinking skills (and the reason why we have to import so many math and science types). This has been a problem for decades, and is getting worse. Some day, our schools will take leadership in letting parents know clearly that allowing too much electronic media saturation is bad parenting; that TV, video games, recreational computer use, etc., will have a lasting negative impact on their children's ability to learn.

@ patrickmattimore1: The reaction of being upstaged in 1957 by Sputnik caused a revolution in science instruction. Many school district planetariums all over the nation owe their existence to America's reaction to Sputnik, & their decline to Nixon's decision to cut funding to NASA from just under 4% of the budget to less than 1%, & his successors continued to nickel & dime the agency. We are still hurting from short-sightedness from over thirty years ago.

"Among the 33 other OECD countries, 12 had higher average scores than the United States..."

"Among the 64 other OECD countries, non-OECD countries, and other education systems, 18 had higher average scores..."

We are significantly behind 30 other contries in science education! Science education forms the basis for much of our national success. We're losing it.

The government gives our grants to programs that have been tried before and failed.

Foundations give grants to non-profits who have opted out of the free market that built our country.

Fund education entrepreneurs!

Next, note Ms. Strauss has commented on NCLB. She's right. Emphasis on tests has made things worse or, maybe, not better. I say worse because maintaining test scores when everyone worries about test results is a sorry result. It's also moved our students even further away from thinking and more toward memorizing.

We CAN fix this problem, but not with political posturing. I have one piece of the solution in hand and successfully deployed to over 25,000 students. We must have more such solutions and more support for the ones already in place.

Is PISA "a Sputnik wake-up" or are international comparisons invalid. Rather than wade into that debate, I'd rather look more closely at the questions in the PISA test and what student responses tell us about American education. You can put international comparisons aside for that analysis.

Are American students able to analyze, reason and communicate their ideas effectively? Do they have the capacity to continue learning throughout life? Have schools been forced to sacrifice creative problem solving for “adequate yearly progress” on state tests?

I focus on a sample PISA question and an insight into what American students can (and cannot do) in my post "Stop Worrying About Shanghai, What PISA Test Really Tells Us About American Students" http://bit.ly/eChNoY

Is PISA "a Sputnik wake-up" or are international comparisons invalid. Rather than wade into that debate, I'd rather look more closely at the questions in the PISA test and what student responses tell us about American education. You can put international comparisons aside for that analysis.

Are American students able to analyze, reason and communicate their ideas effectively? Do they have the capacity to continue learning throughout life? Have schools been forced to sacrifice creative problem solving for “adequate yearly progress” on state tests?

I focus on a sample PISA question and an insight into what American students can (and cannot do) in my post "Stop Worrying About Shanghai, What PISA Test Really Tells Us About American Students" http://bit.ly/eChNoY

Why do we have to measure school success with a test? The egalitarian United States will never match up with elite world high schools on paper and pencil measures of "head" knowledge nor should we. As Dewey said, the "child should be the focus of the curriculum" and helping them become health, well adjusted members of society should be our goal. I am amazed that more people aren't questioning the tyranny of the test.

I would like to know if a research organization has found data that supports the cause and effect relationship between high international test scores and a country's or human's success. I have looked at the PISA and TIMSS (The International Math and Science Study) rankings where the US scores average. But I have also looked at the GNP PPP rankings and US is 6th out of 181 countries. Qatar, which ranks 2nd on the GNP PPP, is ranked in last place on the TIMSS. On patents applied for and received, the US and Japan battle it out at 1st and 2nd place in different categores, while Singapore and Hong Kong are not even on the top 10 list. Finally, there is the Human Development Index (HDI) rankings. This list compares countries based upon 4 indicators: life expectancy, mean years of schooling, expected years of schooling, and gross national income per capita. 1st is Norway...2nd Australia...3rd New Zealand...and 4th US. Where are those countries in terms of the TIMSS? They trailed behind the US in both 4th and 8th grade. I am not saying that we should not continue to look at rigor in our education. We should. I am just questioning whether anyone has taken the time to actually prove that a one shot test score is directly correlated to the future success of a person or country. Maybe there is something to this "teaching the whole child" idea. Finally, I consider myself a successful person. I would hate to see what my 15-year-old self would have scored on a test that I might have believed meaningless to me at the time of taking. Call it the American teenage psyche....but we do grow up, and we do become successful.

IF there really are problems with the US education system, could it be that it has been politically and philanthropically tinkered to death over the last 30 years, and therein lies the problem?

China also discovered a few years ago that the American education system was superior to theirs and made changes. We, on the other hand, changed our system of creative pedagogy and replaced it with an evolution of testing. Tests are snapshots, not albums.

Finally, let direct you to Gerald Bracey's article for an understanding of what international test scores really indicate -- http://www.huffingtonpost.com/gerald-bracey/international-comparisons_b_149690.html

I recommend reading the findings of Yong Zhao, the author of the book, Catching Up or Leading the Way. He brilliantly explains how US education is trying to be more like China and how China is desperately trying to be like the US. He grew up and taught in China and he understands what the US has that few countries in the world have. With our current policies we are undoing those advantages. Our education system has been in a crisis for decades as witnessed by fiction (Blackboard Jungle 1953) to nonfiction (Sputnik 1957 or A Nation at Risk 1983) to NCLB. We can never waste a crisis - it enables more money to be thrown at the problem in our futile attempt to solve it.
Like the War on Drugs, Hunger or Poverty, this war will be fought for decades with a lot of effort, time and money. We will have little to show for it.

I am an American teacher working at an International School (with mostly American students) in Shanghai. Next door to our school is a local Chinese school for very privileged students--children with wealthy families and/or well-connected families. As a colleague said the other day, "Despite the PISA results, I'm still glad my kids go to our school."

We have found that students that transfer into our system from Chinese systems do some things very well. They take tests well, they calculate well, and they memorize and regurgitate facts better than the vast majority of our American students. What they don't do well is think outside the box. Critical thinking evades them.

What many of the articles fail to take into consideration is that many Asian countries track students on an ability level, based on testing that begins quite early. Students who don't pass the tests are done with schooling. They don't stick around in schools long enough for their scores to lower the average, nor are they taking up teacher time that could be spent on higher-acheiving students (I am not at all condoning this method, in fact, I disagree with it--I am simply stating this as a fact). If the US only tested the top 25% of their students, what would the results be?

If the American system is failing so badly, why is it that so many students from so many countries are fighting to get into US schools and universities? Perhaps we're not doing as poorly as advertised.

This is the argument that i have been hearing since i have been in the US since 1966. We are better at "critical thinking" and look everyone comes here. And slowly our students are not competing well with other countries where it matters, job skills. And i would argue that the inverse is what is happening in the US. It's students from other countries coming over with their skills and love for education and then staying that is helping American Universities.
I think we should quit our lazy attitude that somehow because we are Americans we inevitably will do well in everything. It just does not work that way.