Counting, measurement, and the Fraser Report

For those of you unaware of what this means, the Fraser Institute is a private Canadian “think-tank” that concerns itself with education and statistics, attempting to create “a free and prosperous world through choice, markets, and responsibility” by ranking Canadian schools.

Every year, the Fraser Institute releases “report cards” for both elementary and secondary schools, assessing high schools on “diploma examination results, grade-to-grade transition rates, and graduation rates.” All this data is carefully collated and schools are given a score out of ten. My school’s staff room is in a tizzy because our school is not just in the top 20 out of 276 schools in Alberta, but we’ve beat out every other school in our division by 70 points or more.

I looked at what made up this year’s score, and what I’ve discovered has made me laugh, in some cases literally out loud. To put things into perspective, our graduating class last year was 24 students. That’s right: our prestigious score was based on a massive sample size of 24 students. In the top 10 schools, one school had a graduating class of 736, and the smallest had 26. I didn’t study statistics in school, but I do know a thing or two about sample size.

That aside, the reason our school did so well last year was because we had three exceptional students who did extremely well on diploma exams, the final matriculating exam here in Alberta worth 50% of a course’s mark. I went through the data, and the English 30-1 average score went up by – wait for it – three per cent. Three. The average Biology 30 score went up by about five per cent. These gains are repeated through most of the core subjects. Now, in a graduating class of 24, all it takes is for a couple of students to do exceptionally well for these scores to be skewed. If smaller schools can get high results based on small class sizes, how can larger schools compete? If a school is located in a poor neighbourhood in the province, how can they compare with, say, private schools in wealthy suburbs, especially if the differences in these diploma exam results are counted in single percentage points?

The Fraser Institute claims that “[p]arents can use the school report cards to compare the academic performance of schools when choosing a school for their children.” I would contend that the academic performance of our school has remained exactly the same over the past five years, but because our school has so few students, our “report card” fluctuates quite a bit. But a parent seeing these results might wrongly believe that we’re doing something different and special at our school, and remove their students from excellent neighbouring schools and place them at our own.

Can schools really be chosen by diploma exam results? Would any thoughtful parent really take that into account when placing their children at a school? As a parent, I’m more concerned about the atmosphere at a school, whether students are treated with respect and given the freedom to pursue their own learning. I’m more interested in the fine arts programs and the extracurricular opportunities afforded to students. It would seem entirely unfortunate to rate an entire school based on the performance of last year’s graduating class, especially given the tremendous number of variables used to calculate these report card scores. (I did take algebra in high school and as such I’m able to offer the dim recollection that a problem with two variables can’t be conclusively solved; there’s no single answer to the question “x + y + z = 19″.)

At the top of the search page, the following warning is printed in red: “Thomas Lukaszuk, Alberta’s Deputy Premier, has called our school performance reports a misuse of provincial test results. He claims they mislead you and he wants to redesign the tests so we can no longer produce the reports. Our school performance reports are valid and used by many thousands of thoughtful parents each year.” I completely agree with the Hon. Deputy Premier: these reports do mislead parents, and based on what I’ve seen, I don’t know how the Fraser Institute can claim that their score is valid.

Three excellent students were enough to skew our report card upwards some 60 points. That’s all it took. At the top of the report card page, the Fraser Institute opines “If it matters… measure it.” With all due respect to this institution, I completely disagree. William Bruce Cameron’s oft-quoted (and oft misattributed) adage about measurement is applicable here, and I’ll leave it at the end of this musing: “Not everything that can be counted counts, and not everything that counts can be counted.”

Like this:

Related

About alanthefriesen

Discussion

4 thoughts on “Counting, measurement, and the Fraser Report”

Excellent commentary! Two thoughts come to mind for me. One is a cliche I often share with students, family, and friends: “You get what you pay for; if it can be reduced to a simple (single) equation, it’s value should be obvious.” The other thought is one considered by all of us ABET (engineering accreditation) report developers: “There are three components of any assessment: the method used, the outcome(s) from using that method, AND the justification of the effort – why the method was used and what the results mean.”

The pure numerical ranking based upon a single equation without any discussion of method or results is as you note laughable. At the very least, a tabulated uncertainty together with the relative contributions to that uncertainty should be considered a must.

I would be happy if K-12, undergrad, graduate and professional education were given deep consideration in the context of a mere seven years. Perhaps serious consideration should be given this question:

Under the utopian condition of a 100% graduation rate of exceptional students, what are those high school, undergraduate, graduate and professional school grads then to do?

I believe the “West” is chasing its tail, and that education is but a fraction of that chase…

One possible avenue toward a longer view would involve the dropping of fear of losing “We’re Number One” status, nonspecific (yet terrifying) threats, fear of losing SES status and fear that we might actually have to give up somethings in order to benefit “the children”. Better yet, let’s shed the West’s zero-sum game mentality — But that is a cultural matter unrelated to “education” — or is it???