Search This Blog

Who makes it into PISA?

Unlike earlier PISA reports, the 2015 PISA report (Volume I and Volume II) highlights differences in sample coverage – how many students were eligible to participate in PISA – between countries. Why is this important? Because you can’t really appreciate the magnitude of improvement in a country’s performance in PISA unless you also understand how access to education has expanded over time, too.

Take the case of Turkey. Of the OECD countries that participate in PISA, Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores. Between 2003 and 2012, Turkey managed to improve its mathematics score by 25 points while also narrowing the achievement gap between rich and poor students (in other words, improving equity in education). Over the same period, Turkey also managed to keep students in education longer and see them progress more steadily through grades – achievements that, until now, have gone largely unrecognised.

In order to be eligible to sit the PISA assessment, a student must be between 15 years, 3 months and 16 years, 2 months old, still enrolled in school, and in grade 7 or higher. These details may sound like trivial technicalities, and in most OECD countries they are. But in Turkey and Mexico, and partner countries like Viet Nam and Indonesia, these details can make a big difference. This is because early school-leaving, dropout and slow progress through grades are widespread and substantially affect PISA sample coverage.

A recently-released paper shows that in 2003, fewer than one in two Turkish 15-year-olds was eligible to sit the PISA test. Household survey data from the 2003 Turkish Demographic and Health Survey show that 33% of 15-16 year-olds had already left school, and another 18% were still enrolled in grades 1 through 6. Thus the 2003 PISA results were only truly representative of less than half (45%) of the population of Turkish 15-16 year-olds.

But between 2003 and 2012, there was a significant increase in students remaining in school and a considerable decline in excessively delayed grade progression. The figure immediately below, based on data from the Turkish Demographic and Health Surveys of 2003, 2008 and 2013, shows that the percentage of students who were eligible to sit the PISA test increased from 45% in 2003 to 80% in 2012.

The paper explains that if one takes into account these large expansions in access and attainment, the improvement in Turkey is much greater than is usually thought. For example, taking into account both improvement in test scores and the expansion of the population of 15-year-olds who are eligible to sit the PISA test, the paper shows that the increase between 2003 and 2012 in the percentage of 15-16 year-olds attaining baseline Level 2 in mathematics and reading is more than twice as large as previously considered. While this paper only covers the PISA cycles from 2003 to 2012, in PISA 2015 Turkey’s performance declined in the three core domains of assessment, and coverage expanded only minimally compared to 2012.

Although all students benefited from the improvements in Turkey, girls and the poorest 40% of students benefited the most. This was largely because the dropout and early school-leaving rates among girls halved from 38% in 2003 to 20% in 2013. During the same period, the percentage of disadvantaged 15-16 year-olds who attained Level 2 in reading increased more than threefold, from 13% to 46%. Since the percentage of advantaged students who attained Level 2 also increased – from 50% in 2003 to 82% in 2012 – the performance gap between rich and poor remains large.

Analyses reported in the paper also find that advantaged students are considerably more likely to be eligible to sit the PISA test – and to acquire basic proficiency in mathematics and reading – than disadvantaged students.

This new research shows the importance of accounting for who makes it into the PISA sampling frame. After all, survey results are only as representative as the students that make it into the sample. To address the problem of coverage, the OECD is piloting a survey for out-of-school 15-year-olds through the PISA for Development programme. The programme is currently conducting the field trial of this component, which will become available to all countries participating in the PISA 2021 survey.

Comments

Popular posts from this blog

by Dirk Van DammeHead of the Innovation and Measuring Progress Division, Directorate for Education and Skills

The label “21st -century skills” is being increasingly used, and sometimes misused, to indicate that the rapidly changing economic, social and cultural environment of the current century demands a revision of what we think are crucial subjects for the next generations to learn. Examples include creativity, innovation, critical thinking, curiosity, collaboration, cross-cultural understanding or global competence. Some people wonder whether these skills are truly new, or whether education has always been about fostering these capabilities. But stakeholders – not least employers and the business sector – continue to complain that they don’t find candidates leaving the education systems who have the skills they think matter for the jobs they have to offer. And they claim that this is the case because current education systems do not sufficiently prioritise the development of such…

by Andreas SchleicherDirector, OECD Directorate for Education and Skills

Schools nowadays are required to learn faster than ever before in order to deal effectively with the growing pressures of a rapidly changing environment. Many schools however, look much the same today as they did a generation ago, and too many teachers are not developing the pedagogies and practices required to meet the diverse needs of 21st-century learners.

In response, a growing body of scholars, educators and policy makers around the world is making the case that schools should be re-conceptualised as “learning organisations” that can react more quickly to changing external environments, embrace innovations in internal organisation, and ultimately improve student outcomes. Despite strong support for and the intuitive appeal of the school as a learning organisation, relatively little progress has been made in advancing the concept, either in research or practice. This lack of progress partly stems from a lack …

by Andreas SchleicherDirector, OECD Directorate for Education and Skills

Globalisation is connecting people, cities, countries and continents, bringing together a majority of the world’s population in ways that vastly increase our individual and collective potential, and creating an integrated market in products and services. One in three jobs in the business sector now depends on demand in other countries. In fact, a single product is often produced by workers in different parts of the world along the so-called Global Value Chain. Global value chains give small companies and countries unprecedented opportunities to reach global markets and create new jobs.

But the same forces have made the world more volatile, more complex and more uncertain. The rolling processes of outsourcing and the hollowing out of jobs, particularly for routine tasks, have radically altered the nature of work. For those with the advantage of the right knowledge and skills, this is liberating and exciting. In In…