Thursday, August 03, 2006

The New Zealand Living Standards 2004 report depends entirely upon its "Economic Living Standards Index" (ELSI), first used in the previous (2000) report. At that time I expressed reservations about the index. Many have not been addressed. What I do here is set some down again, and then mindful of the ELSI's problems, and try to draw some conclusions of what the latest survey may be telling us about what happened between 2000 and 2004.

This may all seem a bit tedious. Who cares if the measure does not have any real meaning? Is it not better to use it for whatever (political) purpose we want without worrying about what the measure actually means? Don't we do that all the time? Do we need to have a detailed understanding of the meaning of time or income or whatever? Cant we just trust those who constructed the measures?

Well actually no. Typically scientists have spent much effort in constructing valid indexes. They are usually based on sophisticated theory, while each measure is verified by other scientists and is shown to have a practical significance related to things outside the narrow confines of the data which generates the measure. This is so routine for our authoritative measures that we assume it is true whenever someone proposes a new one. It may not be. Often indexes (i.e. measure of something or other) are not validated and amount little more than the opinions of the proponents.

Validation usually involves two stages. First, there is the underlying theory. Is it rigorous? How does the index relate to it? (Its construction often has to make compromises, but if the theory is well understood one can evaluate the extent to which they matter.) Second is a empirical validation. Does the index relate to anything outside the data from which it was constructed? (Two examples of my trying to empirically validate indexes – sometimes called ‘calibration’ – are at http://www.eastonbh.ac.nz/?p=229 and http://www.eastonbh.ac.nz/?p=460.)

Validating the ELSI Index

The ELSI is based on asking households (strictly an "Economic Family Unit" – EFU) a set of questions about what items they have or have not got, what restrictions there are on their social activities, what economising behaviour they have had to practice, and how they rate themselves. The household responses are then combined to give the ELSI index.

Note there are two stages. The first involves the questions asked. Are they the right questions? The second involves the aggregation of the responses. Has the right weighting (significance) been given to each response? How do we know the selection and weightings are not merely the opinions of those who construct the index, and that another set of "experts" would make different decisions? (For instance, the ELSI doubles the significance of some responses relative to others. Why not three times? Why not half?)

As far as I know, there has been little attempt to validate the ELSI. There is no reference to a satisfactory validation in either the report nor its bibliography. That does not mean the index is necessarily invalid, but we need to be most cautious when using it.

In my view the ELSI is a very poor measure of overall economic living standards (whatever that means). In particular, it is unlikely to be much use discriminating between those who are comfortably off. I should not be at all surprised if Bill Gates and myself would get much the same score, as might someone on the average wage and myself. That is because the questions are not designed to discriminate between the affluent. (There is not even a question on car ownership.)

Thus one can give no significance to the average ELSI for the whole public. (If everyone got a(n extra) car it would have to have no effect on their ELSI score – other than perhaps through changes in self-rated satisfaction.)

The difficulty arises because the researchers were focussing on the circumstances of the poor. So while we should dismiss the ELSI for the population as a whole we cannot be so dismissive of the index for those near its economic bottom. It is important if some people do not have a good pair of shoes (supposing that means "suitable for general purposes", rather than "for dressing up"), which was one of the questions.

So what I am going to focus on are those the authors describe as being in some or greater "hardship". That category includes those in "severe hardship" and "significant hardship". In my judgement their samples are too small and the measurement error too big for one to make comparisons involving them

Comparing 2000 and 2004

Famously, Moser's Law says that if a statistic looks interesting, it is probably wrong. Despite the rise in real incomes between 2000 and 2004 (which the report acknowledges) the ELSI decreases slightly. Which is surely "interesting", (I don't have a feel of the significance of the fall in the ELSI from 40.6 to 39.7 – out of 60. That is one of the consequences of not validating the index.) But given that the index is probably meaningless for the majority – say top two-thirds – of the population measures based on the whole population (such as the mean) are of little value.

What is more disturbing is the evidence that more of the population (that at the bottom) were in the hardship category. The rise in the proportion of the population in hardship rose from 23.6 percent in 2000 to 24.0 percent in 2004.

A useful table (C10) in the report, which gives the proportions for the total and a huge range of sub-populations, shows this difference is not statistically significant – that is it could arise from the vagaries of sampling.

There are a few cases among the sub-populations where there are increases and a handful where the difference is statistically significant (but remember that will happen by accident on occasions, given the way that statistical tests operate). You can pour over the table, and find some example that suits your assumptions. (For instance Labour-committed supporters will mention the decrease in hardship of the proportion of the Maori and two parent families. Those on the other side may draw attention to that there seems to be more employed and more of the elderly in hardship.)

At this point we could drop the whole exercise, concluding the changes in the ELSI over the four years tell us nothing. But there is a really interesting problem – one where the "probability" in Moser’s Law warns us sometimes happen. Sometimes a statistic can be interesting and not wrong.

The problem is this. The four years between the two surveys were ones of economic prosperity by almost all conventional welfare measures – higher real incomes and employment, lower unemployment ... – as well as common sense. Yet there is no improvement in the ELSI. The inconsistency is sufficiently strong to suggest that the ELSI is generally not valid. I have already argued that is true, so I am not surprised. At which point we could end the story.

What interests me though, is that where the ELSI might thought to better represent living standards – among those in "hardship" – there is no improvement either (indeed, a statistically insignificant deterioration). I find this much harder to understand. The ELSI cannot be that hopeless.

I tried to think of some technical reasons. Perhaps household fragmentation has had an effect. When a poor couple splits up only is one is likely to have warm bedding (one of the questions asked) at best. Perhaps the drift north has had an effect too (they were asked about having a warm winter coat). But such effects are likely to be trivial in the four year period.

The report acknowledges that beneficiaries had no real increase in their standard benefits over the four years, and some had reductions because some assistance (such as for families) was not adjusted for inflation. Would their lower real incomes be enough to increase their hardship and reduce the ELSI? Possibly.

There is also a tricky problem that the sub-populations change over time. Suppose a better-off beneficiary in 2000 joined the workforce by 2004. That would lower the ELSI of the beneficiaries who are left. It may also lower the ELSI scores of the employed since they ex-beneficiaries are likely to be at the lower end of the standard living of the employed. Bother, bother, bother.

There is another phenomenon which the ELSI does not deal with well. In each survey, respondents were asked whether they had a personal computer (for example). Now the options (to simplify a little) is "yes", "no" (because I cant afford it), or "I don't want one". The "don't want ones" are then netted out. Now it seems possible that some people who said "don't want one" on 2000 said "no because I cannot afford one" in 2004. Their ownership has not changed one iota, but the standard of living of the household measured on the ELSI will have decreased. Are there enough of such items in the survey to actually depress the score? Possibly.

What too about the family that has bought a computer because they now want (or think their children need) one and have made other sacrifices to fund the computer? The way the index works might depress the score. Indeed the additional purchase need not even appear in the survey. A household which bought a car (not in the survey) and sacrificed their annual holiday (in the survey) to do so. That would definitely depress the score. (And to complicate the story further, what if the car was necessary to go to work?)

Trickling Down Prosperity?

At this point one might become so uneasy about the ELSI, to judge of it of no value. However I think there is an interesting conclusion from the previous couple of paragraphs. To put it provocatively, during a time of prosperity, expectations may trickle down faster than income.

This proposition brings together two issues. For more than thirty years I have been advocating that we should use a relative poverty measure. This expectation effect illustrates its relevance.

Second, for some groups – the elderly and children – there is no automatic market trickle down effect, and that sharing prosperity requires some active intervention by the government.

The elderly have a trickle down through the (imperfect) indexation of New Zealand Superannuation to wages. Beneficiaries don't, because their benefit is indexed to consumer prices which rise slower than wages.

Children have suffered even more because their public assistance is not indexed at all. (There is a big lift in the incomes of some of them via the Working for Families package.)

I have argued that the benefit of the rising prosperity to those on social security has been to get an income increase from a job (not all of which disappears in the costs of employment and taxes and benefit abatement). Those on low earnings may also benefit from longer hours, higher pay and better working conditions, and upskilling. As pointed out, the ELSI can miss such improvements.

But I have to modify that conclusion. What the ELSI for low income people seems to be saying is that those that have not got a job have not benefited. Even if their real incomes are maintained, they may experience falling "living standards" if prosperity raises expectations but not their incomes.

The Real Issue

We can go on in this sort of speculation, but the first real lesson is not to trust changes in the ELSI which contradict common sense. However the study affirms one old truth, and challenges a more recent one.

We have long known there are some groups in hardship. According to the survey over 35 percent of our children are (compared to less than 25 percent of the population as a whole, including children). Interestingly, the hardship measure suggest proportions similar to those in the income poverty studies – the vast majority of the poor in income terms or in hardship in this study are children and their parents. (See various papers indexed in http://www.eastonbh.ac.nz/?p=152 for more details.) Perhaps this should be no great surprise, although it is satisfying to a researcher to have a rather different method coming up with similar conclusions.

The degree to which the government's recent measures to increase family assistance will markedly reduce hardship remains uncertain, because the support has been targeted on working families, rather than all families.

Which leads to a challenge to a more recent "truth". I think it was right in the early 1990s to place some emphasis on beneficiaries finding jobs. The conclusion, dimly seen through this report on living standards, is that after 15 years of doing so, plus a period of employment expansion and prosperity, there remains a rump who are still trapped into hardship. The policy strategy, which has succeeded for many, has failed them. If we are concerned about their hardship, and the often serious and longstanding (even inter-generational) deleterious consequences, we need to think about an alternative strategy which may be more income. It almost certainly is not to intensify a strategy which has failed them in the past.

5
comments:

Well, I posted it because firstly, I'd solicited something, and secondly, it was interesting. As for my comments:

WRT questioning the construction of the index, its difficult to see what the alternatives are. This is social science, not physics, so we have no way of measuring poverty to validate the index against; all we can do is construct something that "looks right" and which seems to match what people call hardship. The problem, as shown by your comments on the earlier post, is that some people set far lower standards for what constitutes "hardship" than others...

The problem of increasing expectations affecting scores is noted in the report, and is expected to lead to a slight depression in scores (interestingly, they also used computers as an example, but these are a good where the reverse has happened; expectations have increased, but access has increased even more). This was expected to be one or two points - perhaps explaining the overall decline - but many of the most interesting results (those showing increases in severe hardship) show greater differences than this. I should also point out that the most horrifying results - the increases in severe hardship in general, and specifically among Maori, Pacific Peoples, children, and beneficiaries - also are the statistically most robust, with extremely low p-values. Despite Brian's worry about small sample sizes, most of these results have p < 0.005, meaning that there's a less than 0.5% probability that they would arise by chance (the least significant result is the increase in hardship among EFUs with equivalent disposable income of $10,000 of less, which is only p < 0.02). So, while its fair to question the way the index was constructed and its relation to reality, I think its wrong to question the internal statistical significance of the results. People would kill for p-values that good elsewhere in the social sciences...

I think the conclusions at the end - that incomes at the bottom of the heap have failed to keep pace with the rest, and that there's only so far an employment-centered policy can go - are basically right, particularly when you consider the rise in sickness beneficiaries. These are people who (for one reason or another) cannot get jobs (at least for some period of time) - meaning that there's basically no way out for them other than getting better (to the extent that that's possible). Reducing hardship for this group then requires the government to invest specifically in their medical care - which they are at least currently experimenting with. As for the rest, the government is simply going to have to raise incomes. On the plus side, the increases they've made and plan to make to the minimum wage should at least allow them to argue that this is simply a matter of keeping pace, and that it doesn't change the incentives too much.
Posted by
Idiot/Savant
:
8/04/2006 05:06:00 PM

interesting,

In 2003 I was lucky enough to meet some of the researchers behind the initial index. One point they made which sticks in my mind was that the correlation in income poverty and poor outcomes (as measured by ELSI) was very weak, and that it only became more significant when housing costs were taken into account (even then it was still hazy enough to indicate that other factors were at play). I haven't had time to read the latest survey, but it struck me that the steepish rise in housing prices (and other values) over the years between the two surveys might - possibly - have considered to the worsening of outcomes, in a time of rising incomes.

One otherthing - I vaguely remember Brian Easton berating Charles Waldergranve and Bob Stephens in SPJ for using relative poverty measures. Perhaps my recollections are wrong, but otherwise he has had a change of heart.

The other thing to consider is the weather. Peoples moods are affected by the weather, tending to be more optomistic and satisfied if the weather is good.

The 2000 survey ran from Feb to June, whereas the 2004 survey was from Mar to June.

Notable Weather: Autumn 2000by Trevor McGavin, MetService N.Z. Ltd.

Autumn 2000 was an unsettled season with numerous rain-producing troughs crossing New Zealand. Some troughs produced spectacularly unstable atmospheric conditions resulting in remarkable heavy rain events, especially over northern districts. These included one storm which gave an unofficial record-breaking hourly rainfall rate in Northland on Good Friday (21 April).

Autumn 2004 included a number of storms worthy of note, although by contrast with our exceptionally stormy summer just past, this season was much quieter.

Included in detail here are the winter-like cold snaps of late March and early April, which brought early season snow and frosts, and the heavy rain which affected northern parts in early May.

So depending on which actual days the surveys were carried out, there could easily be statistical differences. Autumn 2004 would appear to have been colder than Autumn 2000, so people could have been feeling their hardship more acutely.
Posted by
Brent
:
8/09/2006 09:47:00 AM

Interesting post & comments I/S. We can conclude that the attacks on Labour for increasing hardships are all caused by perception and the failure of people at the bottom to maintain progress at the same level as other people.

What was most revealing was his comment that the welfare changes in the early 90's had a beneficial impact over the longer term. Certainly an admission that would be anathema to the current government.
Posted by
sagenz
:
8/09/2006 10:38:00 PM