In October 2013, I posted an essay, “PISA’s China Problem,” that called on the Organization for Economic Cooperation and Development (OECD) to fully disclose its arrangement with China regarding Shanghai’s participation in the Program for International Student Assessment (PISA). The latest PISA scores were to be released in December, offering an excellent opportunity for the OECD to dispel the mystery surrounding Shanghai’s 2009 involvement with PISA. I noted that Shanghai, the wealthiest, most educated province in China, was the only mainland province officially participating in PISA 2009 and PISA 2012. Other data from rural areas of China had been talked about by PISA officials over the years, but never released to the public domain. I called on PISA to release those data.

When the latest PISA scores came out in December, nothing had changed. I followed up with a second essay. I again urged full transparency. I also challenged PISA’s portrayal of Shanghai as a “high equity” school system. An extensive literature—including excellent journalism and both qualitative and quantitative scholarship–documents the cruel effects of the hukou system on migrants in Shanghai. Hukou is an internal registration system in China that limits rural migrants’ access to urban public services, in particular, to schools. These migrants are Chinese citizens, mind you, not immigrants from other countries. They have simply moved from rural areas to China’s big cities, or, because the hukou is inherited, they were born in one of China’s big cities but because of their family’s rural hukou, have become second generation migrants in the eyes of the state.

Andreas Schleicher of OECD-PISA wrote a response to my essays, as did Dr. Zhang Minxuan, President of Shanghai Normal University. Marc Tucker, President and CEO of the National Center on Education and the Economy, joined Mr. Schleicher in publishing a third response in Education Week.

I answer them here. I organize the discussion by the two main questions in dispute.

Issue #1: Is Shanghai a Model of Equity?

OECD-PISA has portrayed Shanghai as a model of equity since the province began participating in PISA in 2009. In the foreword to the 2009 volumes analyzing PISA results, Shanghai is the first PISA participant mentioned by name, singled out for attaining high scores and high equity with a diverse population of students:

The education systems that have been able to secure strong and equitable learning outcomes, and to mobilize rapid improvements, show others what is possible to achieve. Naturally, GDP per capita influences educational success, but this only explains 6% of the differences in average student performance. The other 94% reflect the potential for public policy to make a difference. The stunning success of Shanghai-China, which tops every league table in this assessment by a clear margin, shows what can be achieved with moderate economic resources in a diverse social context.[1]

This statement, testifying to the power of public policy, is unintentionally ironical. It ignores the hukou system, a public policy condemned by human rights groups such as Human Rights Watch and Amnesty International, and that China itself labors to change. It ignores how the hukou system has a devastating impact on the lives of migrant children. It ignores how the hukou, by design, shows disdain for social diversity. It ignores how, even today, this cruel but powerful public policy influences who attends Shanghai’s public schools by driving out migrant children.

In their responses to my essays, Dr. Zhang, Mr. Tucker, and Mr. Schleicher spend a good deal of time defending PISA’s sampling procedures. That’s a diversion. I explicitly stated that “this is not a sampling issue.” The issue is the truncated 15-year-old population that remains in Shanghai once PISA sampling commences. In the December post, I presented a table of population data from ten countries and Shanghai showing that the other PISA participants with about the same number of 15-year-olds (Shanghai reported 108,000 in 2012) have national populations that are only one-half to one-third the size of Shanghai. How can a country the size of Portugal, population 10.6 million, produce almost the same number of 15-year-olds as Shanghai, population 23.0 million?

Dr. Zhang calls this approach naïve and unscientific. He raises the valid point that the ten comparison countries are economically developed and most are in Europe.

Table 1 rectifies that shortcoming. It includes all PISA participants except for Lichtenstein, which was omitted because its total population is less than 50,000 (thereby rounding to zero under the table’s rounding protocol), a total of 64 jurisdictions. It includes developed and developing nations and countries with birthrates even lower than China’s. The first column shows each country’s national population. The second column displays the total number of 15-year-olds in the country as reported by PISA. The third column (“15-Year-Olds % of POP”) shows the percentage of 15-year-olds as reported by PISA expressed as a percentage of each country’s overall population. The world average is 1.2872%.

The multiplicative inverse of 1.2872% is approximately 80, meaning that 80 is the divisor one would use to produce a ballpark estimate of the number of 15-year-olds in a country when only that country’s total population is known. The fourth column uses the world average to predict the number of 15-year-olds, the fifth column (Difference) reports how far off the number of actually reported 15-year-olds on PISA is from the predicted number, and the last column expresses the difference as a percentage of the predicted number. This method of predicting countries’ 15-year-old populations is quite robust. The correlation coefficient for the predicted value and that reported by PISA is 0.964.

Table 1 rank orders the PISA participants by the last column. There are two outliers at the bottom of the table, Shanghai and the United Arab Emirates (UAE). These two participants under-report fifteen-year-olds in comparison to all other PISA countries.

Based on the average of all 64 PISA participants, Shanghai’s population of 23.0 million is expected to include 296,060 15-year-olds. The reported population of 15-year-olds (108,056) is 188,004 less than the expected number—or about 36.5% of the expected number. Obviously something extraordinary is going on in Shanghai. The UAE’s status can be explained by the fact that only 15% of its population are citizens. About five out of six people in the UAE are from other countries. It is telling that the country Shanghai most resembles in this analysis is the UAE. But Shanghai has a tiny number of residents from other countries.

Shanghai’s shortage of 15-year-olds is driven by the hukou system. It denies a significant portion of the population, rural migrants, basic rights of citizenship. The missing 15-year-olds fall primarily into two groups: 1) children who are left behind in rural villages and never come to Shanghai with their parents in the first place, and 2) migrant children who are in Shanghai and may even start school there but leave as they get older.

In their Education Week article, Tucker and Schleicher state, “Loveless, as we said, charges that Shanghai runs the Hukou system so as to make sure that migrant families are forced to send their high school age students home so they won’t drag down the high school PISA scores.”

Not exactly. The current hukou system began in 1958 under Mao Zedong, long before PISA. It is a national system with a reach that extends beyond Shanghai and with social consequences that extend well beyond PISA scores.

I do indeed charge that the hukou system drives a significant number of Shanghai’s migrant children back to rural areas to attend high school, leaving a select population to be sampled by PISA. I also charge that the hukou system forces workers who leave rural villages in search of work to abandon their children. An emerging literature is now documenting the psychological scars this leaves on left behind children.[2]

Tucker and Schleicher urge patience, explaining that Shanghai is still “building out” its high school system. Building out? Contrast the “building out” euphemism with this stark description from another OECD report, the OECD Economic Survey: China, published in 2013 by economists within OECD (not affiliated with PISA):

To attend high school it is necessary to take an entrance examination and this must be taken in the locality of registration rather than the locality of residence. In Shanghai, migrant children can only attend vocational high schools. The Shanghai Education Committee justifies local high schools’ refusal to admit the children of migrant workers on the grounds that “if we open the door to them, it would be difficult to shut in the future; local education resources should not be freely allocated to immigrant children” (Ren, 2012). As a result, few migrant children attend general high schools and those who do return to their registration locality find it hard to adapt and often fail to complete the course (p. 91-92).

Migrant children are not just leaving Shanghai’s schools, they are leaving Shanghai. Kam Wing Chan, professor of geography at the University of Washington and a noted expert on hukou and Chinese migration, provided the following graph showing 2010 Chinese Census data for Shanghai. He published a version of this graph in an article critical of PISA in Ming Pao on January 2, 2014 (published in Chinese).[3] The data are disaggregated by age and hukou status. The red bars show children without a Shanghai hukou.

Figure 1. Children in Shanghai (2010 Census)

The number of children without a Shanghai hukou at age 14 is less than 40% of the number at one year of age. The population of migrants begins ramping up again at age 15 as peasants from the countryside pour into the city in search of jobs. These data are cross-sectional and must be interpreted cautiously. But they cast doubt on Dr. Zhang’s argument that Shanghai’s low birth rate or the large number of young, childless migrants causes the peculiarly low proportion of fifteen year olds. These data support the phenomenon that I described in the previous two posts and that Professor Chan documents in his article: an exodus of migrant children from Shanghai as high school approaches. Wealthy migrants, those who are highly-educated and work in Shanghai’s booming financial industry, for example, may find ways of acquiring a Shanghai hukou or otherwise getting their children into Shanghai’s high schools. But for the children of peasants, that option is foreclosed.

As stunning as these data are, they do not include the left behinds. In a June 2013 report from US-China Today, a Shanghai mother explains the dilemma. “I work and live here alone,” says Xiao Cai. “I can’t bring my daughter because her Hukou won’t allow her to get public education here in Shanghai and I can’t afford private tuition.”[5] I asked Professor Chan for an estimate of the number of children left behind by migrants working in Shanghai. He gave a ballpark figure of 200,000 to 300,000.

In his December 10 article, Mr. Schleicher argued that commentators who cite the hukou system as driving out migrant children are bigoted and uninformed:

Others were quick to suggest that resident internal migrants might not be covered by Shanghai’s PISA sample, because years ago those migrants wouldn’t have had access to Shanghai’s schools. But, like many things in China, that has long changed and, as described by PISA, resident migrants were covered by the PISA samples in exactly the way they are covered in other countries and education systems. Still, it seems to be easier to cling to old stereotypes than keep up with changes on the ground (or to read the PISA report).[6]

Note that in the three essays I have written on this topic, almost all of the cited data, scholarship, and journalistic accounts were published no earlier than 2009. Evidence overwhelmingly supports the charge that the current hukou system discriminates against migrants.

Yes, there have been reforms— and Shanghai deserves credit for that— but the effects of hukou continue to be deplorable. The centerpiece reforms were to close the private “peasant schools” serving grades 1-9, to build new schools in neighborhoods populated by migrants, and to integrate migrant students into public schools for grades 1-9. A 2010 China Daily story reported on the post-reform experiences of Wang Zefang, a middle school student in Shanghai:

Of the 300 migrant children who arrived at the Baoshan district school in 2008, Wang was one of a handful admitted into regular classes. The others are housed in a separate school building, wear an alternative uniform and have different teachers and textbooks. Even meal times and finishing times are different.[7]

Professor Pei-Chia Lan refers to this school’s treatment of migrants as “the apartheid model.” In her 2012 study, Lan also describes another model, “the concession model,” and offers as an example a state-subsidized, migrant-specific school.[8] Lan presents a map of the school facilities, drawn with the help of a fifth-grader who is a second-generation migrant student (born in Shanghai but with a rural hukou).

Figure 2. The Concession Model of Shanghai Schools

The black shaded areas are classrooms and playground areas for migrant children, the remaining areas are reserved for local students. Notice that there are even separate entry gates; local students enter through the north gate and non-local students through the south gate. Lan notes, “Migrant students are confined to the leased territory, with second-class facilities, equiptments , and resources. Their teachers, many of whom transferred from shutdown peasant worker schools, receive substantially lower salaries compared with teachers at regular public schools.”[9]

Dr. Zhang and Mr. Schleicher and Mr. Tucker are all correct that Shanghai should be commended for implementing some hukou reforms. Limited as they are, those reforms are significant first steps on what will be a long, difficult road to dismantle the hukou system. But that does not justify PISA’s portrayal of Shanghai as a model of educational equity. There is considerable distance between taking the first steps towards righting an historical wrong and acting in a way that other nations should follow.

It also does not excuse PISA for never mentioning hukou. PISA prides itself on describing the context in which educational systems operate. The PISA reports from 2009 and 2012 do not include the word hukou. Nor do the special 2009 or 2012 PISA volumes on equity. Nor does Marc Tucker’s book, Surpassing Shanghai. Nor does the Shanghai chapter in Strong Performers and Successful Reformers in Education. In that chapter, what we read is this: “In a way, Shanghai has established the notion that migrant children are ‘our children’ and works constructively to include them in its educational development.”

Issue #2 What arrangement does the OECD-PISA have with China?

The 2009 PISA data were released in December 2010. Shanghai-China led the world in all three subjects tested: reading, math, and science literacy. In the publications accompanying the release, Shanghai was identified as the only Chinese province taking part in PISA, a significant fact because of Shanghai’s historical reputation for educating Chinese elites and a per capita GDP twice that of China as a whole. Within days, Andreas Schleicher claimed other Chinese provinces had taken the test and downplayed Shanghai’s exceptionality. “We have actually done PISA in 12 of the provinces in China” he told the Financial Times on December 7, 2010. “Even in some of the very poor areas you get performance close to the OECD average.”

In a 2011 interview with Huffington Post, Mr. Schleicher continued to refer to data from other Chinese provinces, stating, “PISA covers 87 percent of the world economy currently. In China we have only covered 12 provinces so far because it is a very complex place to administer the test.”

In a 2012 BBC report, the unpublished results are again cited by Mr. Schleicher:

Mr. Schleicher says the unpublished results reveal that pupils in other parts of China are also performing strongly. “Even in rural areas and in disadvantaged environments, you see a remarkable performance …Shanghai is an exceptional case – and the results there are close to what I expected. But what surprised me more were the results from poor provinces that came out really well. The levels of resilience are just incredible.”

The BBC report states that the unreported provinces were tested to get an idea of how the whole country of China was doing, also noting that the Chinese government did not allow the OECD to release the results:

Mr. Schleicher is confident of the robustness of this outline view of China’s education standards. In an attempt to get a representative picture, tests were taken in nine provinces, including poor, middle-income and wealthier regions. The Chinese government has so far not allowed the OECD to publish the actual data.

That last sentence is important. As you read this, the Chinese government still has not allowed the OECD to publish the actual data. For three straight years, the story remained the same, with the only mainland province offering public PISA data, Shanghai, portrayed as a model of excellence and equity and references to undisclosed data that showed the remarkable “resilience” and high achievement of disadvantaged students in rural provinces.

Beginning with the October Chalkboard post, I have called on OECD-PISA to fully disclose its testing arrangement with China. The press quotations above are quite unusual and disturbing. I have studied and been involved with international testing for a long time, and I know of no situation in which a government has been allowed to limit the region of sampling and to choose which testing data will be released.

But wait. There is more. In the OECD-PISA responses to my critique, the mystery has deepened. Now there is a different story.

Mr. Schleicher and I both appeared on a December 10, 2013 radio program, Daybreak Asia (you can hear an audio clip below).

I said, “It looks as if the Chinese government first was allowed to pick the provinces where PISA was given, rather than a random sample of all children across China, and then after looking at the scores, the government apparently was allowed to decide which scores would be released.”

Mr. Schleicher responded,

“Actually, that’s complete nonsense. We have an agreement with the province of Shanghai and have taken a representative sample of students in Shanghai. Actually, the sample was drawn by a U.S. company to be accurate and representative.

What is correct is that beyond the Shanghai results, which we report in our international data, the Chinese authorities themselves have been experimenting with PISA in other provinces, but, you know, those data are not representative so we don’t use them in our comparisons.”

And then in Education Week on December 26, 2013 Marc Tucker and Mr. Schleicher said this:

First, there is no secret deal. Shanghai is the only province in mainland China that volunteered to fully participate in the last two rounds of PISA. Other provinces have recently agreed to participate, but they won’t be surveyed until 2015. Sure, other cities and regions in China have been experimenting with the PISA instruments, as is the case in many countries, but none of these entities have sought to participate in PISA, which would have involved drawing representative samples, compliance with the international technical standards, and collaboration with the OECD and the contractors charged with the implementation of PISA. Even though results from these experiments have been widely discussed among researchers and educators in China, it would have been irresponsible to report results from those provinces in the PISA findings.

Notice the word changes. The “we” who conducted PISA in 12 of the provinces has now become “the Chinese authorities” conducting the tests. The rural provinces that were selected to “get a representative picture” are now “not representative” despite the fact that for three years, they and the data drawn from them were representative enough for several public references to the results as if they were. It would now be “irresponsible” to officially report the results, but the results have been discussed extensively in media interviews. And what about the Chinese government not allowing the OECD to publish the rural results? Not a word.

Recommendations and Conclusion

These convoluted explanations place PISA’s integrity at risk.

I now elaborate upon my previous recommendation. The PISA Governing Board, the governing authority responsible for PISA’s adherence to basic standards of assessment, should commission an independent panel to investigate and report on the OECD-PISA’s arrangement with China. Evidence should include: all solicitations of participation in PISA that were made to Chinese provinces–and their responses; all correspondence and agreements regarding sampling and the reporting of PISA scores, whether the agreements were made with the central government in Beijing or provincial authorities; and a full review of the substance and quality of data that were collected throughout China in 2009 and 2012. All data previously collected should be released so that independent scholars can conduct secondary analyses and verify conclusions publicized over the past several years.

In the meantime, the ever changing story of PISA’s special arrangement with China continues to cast doubt on the trustworthiness of PISA.

A December 18, 2013 CNN interview featured Andreas Schleicher and two other guests. The discussion took place in Shanghai. Mr. Schleicher stated, “There is no question that rural schools in China do better than similar schools almost anywhere else in the world.”

The secret rural data are alive again! And, we are told, although we will never have a chance to see them, that there is no question what the data reveal.

The only Chinese educator on the panel objected to Mr. Schleicher’s assertion and argued that rural schools in China are under-resourced and in terrible condition. Mr. Schleicher would not hear of it, responding, “If you are at a disadvantaged school in China, your chance to succeed is much greater than in much of the rest of the world.”

Shanghai is back to being representative of China, the secret data have been resurrected, wild assertions about overcoming disadvantage in China are being promoted, and, once again, hukou and the plight of migrant children go unnoticed.

China has a long way to go in reforming hukou. And the OECD has a long way to go in reforming PISA.