Fighting for Equity in Education

The struggle is long but hope is longer

Money Matters in Education

Sunday July 27, 2014

This is the text of a research brief published by SOS. It shows that money matters in education especially for disadvantaged students. The research supports the recommendation of the Gonski review of school funding to increase funding for disadvantaged students and schools. The references are provided separately below together with a downloadable version of the research brief.

One of the major criticisms of the Gonski funding plan is that it failed to provide sound evidence that increased funding would lead to better student outcomes. It was widely claimed that research evidence shows that the relationship between per-student spending and student performance is weak [National Commission of Audit 2014a, 2014b, Ergas 2014, Justmann & Ryan 2013, Prasser & Tracey 2013, Public Policy Institute 2012, Sloan 2012].

For example, the National Commission of Audit report claimed that there is “no clear, consistent correlation….between increased funding and school outcomes” [National Commission of Audit 2014b, p. 265]. An advisor to the Federal Minister for Education has claimed that the “evidence clearly shows that just throwing more money at schooling makes no difference to student outcomes” [Prasser 2013].

The research most commonly cited in support of such claims is a review of research studies by Professor Eric Hanushek from Stanford University. His original paper is still one of the most widely cited papers in the economics of education [Hanushek 1986; see also Hanushek 1989, 1997 & 2006]. His conclusion “has had great impact and made economists reluctant of increasing education budgets” [Webbink 2005, pp. 536-538].

Hanushek summarized the results of 38 studies and tabulated 147 estimated coefficients on the main expenditure variables according to the effect and statistical significance. He found “no strong or systematic relationship between school expenditures and student performance” [p. 1162]. In a later review he made use of 377 estimates from 90 studies and came to the same conclusion:

There is no strong or consistent relationship between school resources and student achievement. In other words, there is little reason to be confident that simply adding more resources to schools as currently constituted will yield performance gains among students. [Hanushek 1997, p. 148].

What is largely ignored, however, is that Hanushek’s finding was refuted by re-analyses of his reviews. One re-analysis by academics from the University of Chicago criticised the methodology he used to synthesize results across studies as having “low statistical power” which made his conclusion “particularly suspect” [Hedges et.al. 1994, p. 13]. It adopted a more powerful analytic method and found that the vast majority of studies with statistically significant effects show a robust positive relationship between expenditure per student and student achievement. It concluded: “...we find that money does matter after all” [p.13]. The statistical analysis implied that an increase in expenditure of $500 per student (approximately 10 per cent of the national average at the time) would increase student achievement by 0.7 of a standard deviation, which is a very large effect. As the authors stated:

By the standards of educational treatment interventions, this would be considered a large effect. [Hedges et. al. 1994, p. 11]

Another re-analysis of Hanushek’s reviews also criticised his methodology because it placed a disproportionate share of weight on a small number of studies that frequently used small samples and estimated mis-speciﬁed models [Krueger 2003]. This study re-analysed the data on the class size studies in Hanushek’s reviews and arrived at the different conclusion that smaller class sizes (which involve greater expenditure per student) have a positive effect on student performance.

In a later synthesis of some 60 studies, the Chicago University researchers confirmed their previous finding that increases in expenditure has a significant positive effect on student achievement.

The analysis found that .a broad range of resources were positively related to student outcomes, with ‘effect sizes’ large enough to suggest that moderate increases in spending may be associated with significant increases in achievement. [Greenwald et.al. 1996, p. 361]

However, the effect size was 0.14 of a standard deviation which was somewhat smaller than the estimate in their re-analysis of the Hanushek study. It was derived from a more reliable, higher quality and more recent data set than the earlier studies.

Later reviews of research studies arrived at similar conclusions. One comprehensive review and analysis of 35 years of research studies concluded:

A large and growing body of research – that has taken advantage of improvements in technology, better databases and advances in methodologies and measurement – provides further evidence that school inputs can and do make a difference in education and are positively associated with both enhanced student achievement and labor market earnings.
...there are clear relationships between funding and achievement emerging from the recent body of production function research. These studies provide further evidence that money matters in producing educational outcomes. [Verstegan & King 1998, pp. 245, 262]

Another meta-analysis of 46 studies concluded that expenditure per student has a signiﬁcantly positive impact on test scores [Dewey et.al. 2000].

A number of more recent studies have also found a positive relationship between funding and student achievement thus confounding the common view [see Levacic et.al. 2005, Downes et. al. 2009, James et.al. 2011, Baker 2012, Nicoletti & Rabe 2012, Gibbons & McNally 2013]. In some studies, the size of this effect is small while in others it is larger; in some the effect is larger in primary schools than secondary schools; in some additional funding appears to matter more for some students than others; and in others some forms of expenditure have a greater effect than others.

A recent study of the impact of school finance reform in Michigan found that an increase in school funding of twelve per cent per student per year during grades four through seven experienced a four percentage point increase in the probability of enrolling in college, and a 2.5 percentage point increase in the probability of earning a degree [Hyman 2014]. Earlier studies of school finance changes in Michigan also found that increased funding resulted in significant improvements in student achievement [Papke 2005, 2008, Chaudhary 2009, Roy 2011]. In particular, Roy found that an additional $1000 in expenditure per student led to modest to large increases in 4th grade reading and mathematics scores.

A recent study of school funding changes in Massachusetts found that increased funding and a new funding formula that allocates more funding to higher need school districts resulted in significantly higher student performance [Nguyen-Hoang & Yinger 2014]. Student achievement increased in both high-need and low-need districts. It concluded: “Money does matter in raising student performance in Massachusetts, as in other states” [p. 313].

An early study of the changes in school funding in Massachusetts found that the increased funding led to a substantial increase in average 4th grade test scores but not in 8th grade [Guryan 2001]. It attributed the latter result to the fact that 8th grade students had spent less time in well-funded schools. The increased test scores 4th grade students came as a result of improvements by low-scoring students. A later study of the changes over a longer period found that they resulted in large increases in student achievement in previously low spending school districts [Downes et.al. 2009].

A recent Norwegian study also found that increased expenditure has a strongly positive effect on student achievement [Haegeland et.al. 2012]. It also examined resource effects across the achievement distribution and found that the effects were significant for the least as well as for the most able students.

There have also been a number of cross-country studies on the relationship between school funding and international test scores. These have produced mixed results with some finding that increased funding leads to improvements in student results and others finding a weak relationship. For example, Lee & Barro [2001] found that across and within countries more school resources raise students’ scores on international tests. As nations devote more resources to their schooling systems, they simultaneously raise the quantity and the quality of schooling. On the other hand, others find no such relationship [for example, OECD 2013, pp. 40-42, Hanushek & Woessmann 2011].

One of the problems with such cross-country studies is that they use highly-aggregated data. As one study that found no relationship between school funding and student achievement stated, such cross-sectional analysis has to be interpreted cautiously even when controlling for a large set of factors [Hanushek & Woessmann 2011]. There may be reverse causality, and unobserved country differences such as cultural traits or institutional and political factors that may be correlated with both inputs and outcomes.

Few studies have been done on the relationship between school expenditure and student achievement in Australia. One recent study found that additional per student expenditure has a modest positive impact on student test scores [Cobb-Clark & Jha 2013]. The study found substantial gains in student literacy between Years 3 and 5 are linked to additional expenditure on ancillary and expert teachers. Expenditure on ancillary teachers is also associated with significantly larger gains in reading scores between Years 5 and 7.

An earlier widely-cited study found that between 1964 and 2003 spending on schooling increased by 258 per cent in real terms, while reading and numeracy outcomes declined [Leigh & Ryan 2008, 2011]. However, other evidence shows that school outcomes have not declined or have increased over this period, and that the increase in expenditure is significantly over-estimated.

Another study using the same data source found no significant change in reading and numeracy achievement between 1975 and 1998 and improvements for some groups of students [Rothman 2002].

The results reported here indicate that the achievements of Australian 14-year-olds in reading comprehension and mathematics have remained constant during the period. For some groups, there has been improvement, most notably for students from language backgrounds other than English. [Rothman 2002, p. ix]

The two studies used results for different cohorts of 14 year-old students. Leigh & Ryan used the results for 14 year-old students in Year 9 whereas Rothman used the results for all students aged 14. Rothman used the results for all 14 year-olds to avoid bias in his findings because grade repetition was much more common in the 1960s and 1970s than in the 1990s. Leigh & Ryan noted that their findings were sensitive to their choice of students [2011, p.111].

Rothman also considered it a success that reading and numeracy results remained stable over a period which saw dramatic increases in enrolments of students from South East Asian and other countries and for whom English was not the main language spoken.

With such dramatic changes in their clientele, schools found it necessary to ensure positive educational outcomes for a wider range of students from language backgrounds other than English. The data presented in this report show that Australian schools have been successful in providing educational opportunities and achieving positive outcomes for many of these students, reducing differences in scores between students from English-language backgrounds and students from other-language backgrounds, as measured at the student level and at the school level. [Rothman 2002, p. 39]

One problem with both studies is that they used a limited measure of school outcomes, namely, the reading literacy and numeracy results of students aged 14. Another important measure of school outcomes is the proportion of students who stay on to Year 12. By this measure, student outcomes from school have improved markedly since the late 1960s. Between 1969 and 2003, the retention rate from Year 7/8 to Year 12 more than trebled from 27.5 to 75.4 per cent [ABS 1980, 2004].

In addition, surveys of adult literacy and numeracy consistently show that adults who left school in the 1960s and 1970s have much lower literacy and numeracy skills than adults who left school in the 1990s and 2000s. For example, 20 per cent of adults aged 55-64 in 2011-12 had low literacy skills and 29 per cent had low numeracy skills compared to ten and 16 per cent respectively for the 25-35 year age group [ABS 2013; see also ABS 2008]. Also, much higher proportions of the younger age groups have advanced literacy and numeracy levels. All this suggests that literacy and numeracy outcomes have improved over the longer term.

A significant component of the increase in real expenditure was devoted to improving school outcomes that are ignored by Leigh & Ryan – increase in retention in the senior secondary years and increased enrolments of students with disabilities in schools. The huge increase in retention rates to Year 12 was highly expensive because of the wider range of course options available in Years 11 & 12 and the lower class sizes in these subjects.

Similarly, students with disabilities attract much higher levels of funding than other students and their enrolments increased significantly during the last part of the period analysed by Leigh & Ryan. For example, the proportion of students with disabilities in NSW government schools more than doubled between 1992 and 2003, increasing from two per cent of total enrolments to 4.8 per cent [(SCRGSP 1998, 2003]. In Victoria, students with disabilities increased from 1.8 per cent of total enrolments in 1992 to four per cent in 2003.

Furthermore, the increase in real expenditure estimated by Leigh & Ryan is highly sensitive to the measure of cost/price increases. The study provides three estimates of the increase in real expenditure which show a very large variation. For example, the estimates for the increase total real expenditure over the period 1963-64 to 2002-03 vary from 76 to 333 per cent. Their preferred estimate of an increase of 258 per cent is based on a price index that is derived from several sources that measure price increases for different goods and services and therefore it may not accurately reflect cost/price increases in school education.

Apart from the more general studies that show a positive relationship between school expenditure and student achievement, there is extensive evidence that money matters particularly in reducing education disadvantage. Many studies in the UK, the US and elsewhere have found that increased school funding for low SES students leads to better school results [Ooghe 2011; Henry et.al. 2010; Jacob & Ludwig 2008; Papke 2008, Levacic et. al. 2005, Card & Payne 2002].

One UK study that found a small positive impact on student achievement from increased expenditure per student also found a much stronger effect on achievement of junior secondary students eligible for free school meals [Levacic et.al. 2005]. The effect of expenditure per student on students eligible for free school meals was about three times as much as that for non-eligible students for maths and about four times greater for science.

A companion study that examined the impact on student attainment on the General Certificate of Secondary Education (GCSE) found that marginal increases in expenditure improved overall GCSE attainment for all students, but particularly for students from the bottom 60 per cent of the prior achievement distribution [Jenkins et.al. 2006]. It concluded that the findings of both studies “suggest that targeting additional resources at particular subjects or groups of pupils would be more effective than across-the-board increases in resourcing” [p. 44].

A review of research studies on the relationship between expenditure on schools and education outcomes published by the UK Office for Standards in Education shows that numerous international studies conducted since the early 2000s show a positive impact of increased expenditure in schools, especially for disadvantaged students [Gibbons & McNally 2013]. Its strong conclusion is that increases in resourcing are usually more effective in disadvantaged schools and/or on disadvantaged students: “it is more efficient (as well as equitable) to target resources at these students” [27].

Another review of some UK studies found evidence that increasing school resources improves results and that more targeted spending benefits students from disadvantaged backgrounds.

.....this research suggests that increasing school expenditure improves attainment and that it is more beneficial for disadvantaged groups (at least on average). It suggests that targeting resources on disadvantaged groups might be beneficial for helping to reduce inequality in educational outcomes. [Machin, McNally & Wyness 2013, p. 157]

Two recent UK studies of expenditure in primary schools found positive effects from increased expenditure on schools, especially for disadvantaged students. One study examined the impact of increased expenditure from the early to late 2000s on student achievement at the end of primary school using data for all schools in the UK [Holmund et.al. 2010]. It found evidence of a consistently positive effect of expenditure across English, maths and science. The effect size was small, corresponding to about a 0.03-0.05 standard deviation increase in attainment for an extra £1,000 per student. However, the effect sizes were 50 to 100 per cent higher for disadvantaged students.

The other study found much larger effects. It found that urban schools on either side of Local Authority boundaries with high proportions of disadvantaged students receive different levels of funding and that this is associated with a sizeable differential in student achievement at the end of primary school [Gibbons et.al. 2011]. For example, an extra £1,000 of spending was equivalent to moving 19 per cent of students currently achieving the expected level (or grade) in maths (level 4) to the top grade (level 5) and 31 per cent of students currently achieving level 3 to level 4 (the expected grade at this age). The effect size was about 0.25 standard deviations which is a significant effect.

A recent US study found that increases in per student spending, induced by court-mandated school finance reforms, led to significant increases in the likelihood of graduating from high school and educational attainment for low income students [Jackson et.al. 2014]. The spending increases thereby narrowed adult socio-economic attainment differences between those raised in low income and affluent families. The study found that a 20 per cent increase in per-student spending each year for all 12 years of school is associated with nearly one more year of schooling, 25 per cent higher earnings, and a 20 percentage-point reduction in the annual incidence of poverty in adulthood. The magnitude of these effects is large enough to eliminate the high-school completion gap and years of educational attainment gap between children from low and high income families.

Nobel Laureate, James Heckman, together with various colleagues, has published several studies which demonstrate that large benefits are derived from increased expenditure on disadvantaged students in early childhood [for example Heckman 2011; Heckman 2008, Heckman & Masterov 2007].

Even studies that find a weak overall relationship between funding and general student outcomes find bigger effects for disadvantaged students. One such Danish study found that the effect of raising school expenditure on low income students is about three times as high as for the average student [Heinesen & Graverson 2005].

Findings from the OECD’s PISA studies show that targeting resources to disadvantaged students and schools is a feature of successful education systems:

Allocating resources to where they can make the greatest difference is key. PISA finds that the degree of equity with which resources are allocated to socio-economically disadvantaged and advantaged schools is closely related to the performance of education systems. [Schleicher 2014, p.22]

One of the problems with highly aggregated studies of the relationship between overall spending and student outcomes is that they cannot distinguish how the money is spent. As a result some studies show positive results, large and small, while others show no effect or negative impacts. The outcomes really depend on how the money is used and the nature of the complex interaction of different resources available to schools that is not captured in most statistical models. When these complexities are considered the conclusion is emphatic:

…the conclusion that ‘school resources do not make a difference’ is quite wrong, then, and has been the result of studies that are weakly conceptualized and dependent on impoverished data.

The conclusion that money doesn’t matter is not correct, and fiscal conservatives and opponents of fiscal equalization should not take comfort from the complexities of an improved approach to school resources. [Grubb 2011, pp. 8-9, 88]

While many studies find that money matters in education, it is generally acknowledged that the extent to which it matters is heavily dependent on how effectively it is used. For example, one early meta-analysis that found a positive relationship between funding and student achievement observed:

We do not argue that money is everything. How we spend the money and the incentives we create for both children and teachers are equally important. [Greenwald et. al. 1996, p. 385]

Similarly, the Deputy Director for Education at the OECD has said: “How resources are allocated is just as important as the amount of resources available” [Schleicher 2014, p. 24]. Consequently, the focus should be less on whether money makes a difference but better understanding the ways in which money and other school resources can be brought to bear most effectively [Plecki & Castaneda 2009, Grubb 2011].

Clearly, the critics of the Gonski funding plan have not read the research on the relationship between expenditure on schooling and student achievement, particularly in relation to results for disadvantaged students. The claim that increasing expenditure on schools makes no difference to student results is plainly wrong. In particular, the research emphatically supports the central recommendation of the Gonski report to increase funding for those most in need. However, as many studies have also noted, how the money is spent is also critical. Full implementation of the Gonski model should also draw on the wealth of research evidence about the most effective strategies for schools serving disadvantaged students.