A Rank Ranking Rankles

The Age ran a feature on Saturday on suburb liveability. They performed an analysis of each suburb (in conjunction with Tract Consultants and Deloitte Access Economics, who probably did most of the analysis) in terms of various criteria, and then created a suburb league table based on the results. These kinds of analysis are always a little broken, but I was curious about the approach they had taken and, of course, where Dandenong would figure in the rankings.

As it happens, not highly. It makes it into position 232 out of 318 suburbs, putting it around the 27th percentile. That made me a little cranky, but I'd like to be clear why. Such rankings are always highly sensitive to the criteria and methods being used, and there is a fair bit of luck involved in how the ranking works out. Without too much tweaking of the criteria, a suburb could jump between 250th and 50th. What's more, this kind of ranking does not take into account the magnitude of differences: a tiny difference and an enormous difference can both result in a change of just one position in the overall ranks. In other words, these rankings have very low information on their own; they only begin to be informative when you dig into the details. Nonetheless, the media (including, sadly, The Age in this case) treat them as providing moderate to high information.

Now, with a bit of luck, your suburb ends up in a respectable position, and gets completely ignored. With a bit more luck, it ends up ranking more highly, which acts as a good advertisement for the suburb (if that's the kind of thing that you want). Otherwise, your suburb ends up at the 27th percentile, reinforcing other people's negative prejudices against it. Thankfully, in reality, the few people that do look at rankings like this would look at 1) the top 5-10 suburbs, 2) maybe the bottom few and 3) their own suburb, and completely ignore the remaining 300+ suburbs. That's why I was only a little cranky.

In reality, I was more curious than anything. So, I thought I would dig deeper into the details of the study to see if there were areas Dandenong could improve on. I was unable to find any report or source for the data, either via The Age's articles or via a web search. However, I did find The Age's interactive 'Find your Perfect Suburb' calculator. It turns out that some of the source data was embedded in the JavaScript of that page. Here's a little glimpse of what that data looks like in spreadsheet form (with some very light modifications):

Columns F through T correspond to the individual criteria for the suburbs. At first, I assumed each criterion had been assessed on some scale or other (e.g. quantities, distances, etc.), so I calculated the average for all the columns, only to find out that all but two of the columns (Bus and Tram) average to 161. So, no scales here — instead, each suburb has been assigned a rank for every criterion. This includes the Bus and Tram columns, but their ranking has been squeezed down, effectively giving them lower weight. (Which is a really bad way to do the weigh, by the way.) This would be really unusual if these ranks were being used in the weighted sum for the overall rank, so I speculated I actually had the wrong data for reproducing the overall ranks. Nonetheless, I tried summing (just with equal weights) the criteria ranks to see what came up — and I got sums that produced the exact ranking that was used in The Age feature articles. Wow!

Now, sure, this is just a fluffy Saturday conversation piece, intended merely to tickle the suburban pride of the reader. (And in my case, mission accomplished.) And to The Age's credit, they allow people to at least fiddle with the weightings across the criteria using the calculator. But I do wish they had tried a little bit harder to produce something informative and less misleading. For example, by using ranks for every criterion, all of the magnitudes have been eliminated. Why is this a problem? Let's suppose I finished 2nd in a race that was made up of a group of ordinary people plus Usain Bolt. The difference between 2nd and 3rd is in no sense comparable to the difference between 1st and 2nd. And yet, they receive equal weight. As another example, every criterion is treated equal (aside, oddly, from Bus and Tram). I understand that we perhaps don't have a good sense of what these weights should be, but we surely know some things; for instance, that proximity to trams (for example) wouldn't rate nearly so high as lack of crime. Even better would be to survey people and find out what relative weight they would assign to each criterion. Granted, that requires effort, but it's absolutely required before one can start reporting on the results of the overall ranking with any measure of credibility.

To be honest, it would have been far more useful to throw out the overall rankings, and just show the rankings in each category for each suburb. These are Dandenong's:

See, this is already much more informative. (There are other forms I'd like to see this data in, but I think this is passable for a broad audience.)

All the same, there are still problems with this data. There are some awfully questionable rankings here. Let's take the 'Tram' category. The question for this criterion is "Is it important that you live near trams?" and the measure (as indicated in the JavaScript labels) appears to be "Proximity to trams". Dandenong rates 177th out of the 318 suburbs. Wait, what? Dandenong doesn't have any trams, nor is it within walking distance of any trams. (The closest tram stop I could find via a quick Google Maps search is in Vermont South, at about 14.8km straight line distance from Harmony Square. I guess that's an option for some.) Nonetheless, this counts in Dandenong's favour against suburbs like Carrum, Hallam, Langwarrin and Skye. Outside of the tram network footprint, this is a non-measure. Of course, outside this footprint, the suburbs that would rate most highly on this non-measure are — unsurprisingly — suburbs closer to the CBD. But there's already a criterion for proximity to the CBD. Nevermind, the method double-counts it.

The CBD proximity measure is a bit of a problem all of its own. They mention in one of their articles that they don't include affordability because they (in part) want to explain the pricing of houses. (In which case, they probably shouldn't be using the word liveability, since as far as I'm aware, debt affects one's ability to live somewhere.) But the "CBD" is mostly a bundle of many of the other criteria — not just trams, but trains, buses, shops, eateries, schools, etc. This is all double-counting, albeit one that could have been mitigated had better weightings been applied.

There are many other oddities here, but I just want to point out some which are particularly puzzling with respect to Dandenong. Our suburb ranks particularly poorly with respect to culture ("Size of cultural sector"). Goodness knows what "culture" means (art? nationalities?), and lest I make an uncultured fool of myself, I'll restrict my comparisons to the immediate locality. Dandenong ranks 289th. Guess which suburbs rank substantially higher? Dandenong North (at 266th) and Dandenong South (at 232nd). Eumemmerring manages to crack into the top 200 (at 190th). None of these neighbouring suburbs have anything like the scale of arts and/or cultural activity that exists in Dandenong (which has The Drum, The Castle, Heritage Hill, Walker Street Gallery, the library, Harmony Square, etc.) Of the Dandenong and district suburbs, it looks like Cranbourne ranks the highest, managing an impressive 79th. Well, it would be impressive, if I had any idea of what this mythical measure actually refers to. One thing it clearly doesn't refer to is, you know, culture, given that Greater Dandenong is Victoria's most multi-cultural city.

Let's set aside all this criticism, and do what I was originally intending to do— to look at the data for ways in which Dandenong could improve. Dandenong's worst ranking is for crime. It ranks 317th, being beaten only by the Melbourne CBD itself. I completely accept this result. This is a difficult one for the city, because it is the largest urban area outside of the Melbourne CBD (inside Greater Melbourne), while also being one of the most socioeconomically disadvantaged areas, and (I believe) holding the largest refugee and (possibly) migrant populations. The analysis doesn't measure crime rates against (non-private shopping centre) business activity density. I feel like this would show Dandenong's crime rate to be merely well above average, rather than dramatically above average. Nonetheless, we need to do better. At some point, I'd like to review the council's safety strategy to see if it has any teeth.

Another area in which the city does poorly is tree coverage. Again, I agree quite strongly, but I feel like the council is doing quite well to address this problem. Over the past 4-5 years, trees have been planted on most of the main roads. There are some problems with establishment: the trees along the eastern side of Stud Road, for instance, appear to have taken a battering, with perhaps only 70% of the trees remaining. Trees planted in other areas seem to have suffered similar fates, or are otherwise struggling. In itself, this doesn't worry me — more of a worry is that the council doesn't seem to be replanting them. I also worry that council will replant in these places with different tree species (something that was, quite frustratingly, done in Allan Street next to Mosaic) — this is what led to the whole ugly, incoherent, insignificant mess of trees we have today. Still, things are vastly better than they were 10 years ago.

Finally, while we don't seem to do too poorly with respect to open space and eateries, I think we can do much better on both counts without too much effort. On the open space front, we should be making much better use of the city's main waterway, Dandenong Creek, which could be an exceptional asset. With respect to eateries, we could develop subsidies and incentives for people to establish more eateries within the city, particularly ones that aim to provide a bit of an experience or atmosphere in addition to the food itself. We have all the ingredients to make this work extremely well, we just need to find the right recipe.