Overview of the results Why did some people get the Census Test question wrong?- The false negatives - The false positives

Comparison of the follow-up survey and the Survey of Disability, Ageing and Carers- Disability and handicap rates - Comparison of disabilities and impairments

Use of synthetic estimates to predict disability

Conclusions and recommendations

INTRODUCTION

Before each Census, the ABS consults widely with the community and users of Census data about the topics which should be included.

In February 1993, a publication entitled 1996 Census of Population and Housing: ABS Views on Content and Procedures was released. It outlined ABS views on the content and procedures to be used in the 1996 Census and invited submissions about the views expressed in the publication. In response to the invitation the ABS received a number of submissions expressing disagreement with the initial ABS recommendation to exclude the disability topic from the 1996 Census.

This report explains why and how a Census question on disability was tested and analyses the results obtained from the test including the quality of the data obtained.

BACKGROUND

There were 48 submissions initially received, supporting the inclusion of a question on disability in the 1996 Census. It is obviously an important topic and one which is included in the censuses of many countries. However, based on the poor quality of data collected in censuses in the UK, Canada and the USA, as well as in the 1976 Australian Population Census, it had initially been recommended that the topic should not be included in the 1996 Census. Further discussions on Census content and strong lobbying by the Department of Human Services and Health (HSH) led to a reconsideration of the possibility of including a question to measure disability in the 1996 Census.

It was finally agreed that the Australian Bureau of Statistics (ABS) should test a suitable question on disability and then try to measure the quality of the data obtained, by comparing the results with data obtained by personal interview. These results could also be compared with those obtained from the 1993 Survey of Disability, Ageing and Carers (SDAC).

From investigating experiences overseas, it seemed that the main areas which caused problems in data quality were for those where a person only had a marginal disability. Therefore, it was decided to concentrate the Census Test question on identifying those with a severe handicap, in order to try to achieve data of good quality, which would be useful for the various government departments and researchers who required data for small areas and small population groups.

The initial questions were designed by the ABS Welfare Section, as this area has responsibility for Disability and Ageing surveys and therefore understands the concepts and problems involved in trying to measure disability. The first version of the Census Test questions is shown below:

11 Does the person have any long-term physical or mental health condition which has lasted or is likely to last ( ) Yes, has disability >Go to 12 for six months or more ? ( ) No >Go to 13

12 As a result of this condition, does the person need or receive, help or supervision with the following activities: ∙ Moving around at home or elsewhere (for example, ( ) Yes, needs help or supervision getting out of bed, shopping, visiting people) moving around

Extensive consultation was undertaken with interested users, as well as with organisations representing people with disabilities, to determine if these questions would meet the needs of those people requiring the data to be collected.

A number of modifications were made to the questions and the final version used in the Census Test of November 1993 was as follows:

11 Does the person have any physical, intellectual or sensory disability, brain injury or mental illness which has lasted, or is likely to last for six months ( ) Yes, has disability >Go to 12or more? ( ) No >Go to 13 ∙ If the person is less than five years old, leave blank.12As a result of the above disability, does the person ever need, or receive, help or supervision with the following activities: ∙ Moving around at home or elsewhere (for example, ( ) Yes, needs help or supervision getting out of bed, shopping, visiting people) moving around

Although question 12 in the final version remained relatively unchanged, question 11 in the final version was now much more complex. However, the final version of question 11 was expected to provide a better measure of disability, whereas the original version of question 11 would have only measured people with a long-term health condition, not all of whom would have a disability. The original version of question 11 could also have missed people with certain specific disabilities such as loss of limbs, sensory problems (ie loss of sight or hearing) and head injury.

Some concerns were expressed that people would not fully understand what the questions were asking, as these questions were more complex than most other Census Test questions. An analysis of the complexity and structure of the original and final questions using the 'Gunning Fog Index of Readability' rated the original version of question 11 with an index of 12 and the final version with an index of 18. These indexes relate to the number of years of education considered necessary to understand the sentence. For questions other than disability in the Census Test the index was around 6, with the most complex having an index of 11. Question 12 in both the initial and the final version was rated with an index of 14.

Because of the possible problems involved in understanding the disability questions, some observational testing was undertaken in both Canberra and Melbourne. This testing showed that, while people did not necessarily fully understand the first question on disability in the Census Test, they focused on the word 'disability' and provided that they recognised or acknowledged that they had a disability, would then proceed to answer the second question, which would provide the more important data.

A follow-up questionnaire was designed jointly by Welfare Section and Population Survey Development Section (PSD) based on the initial screening questions in the SDAC and the questions used to identify level of severity of handicap in this survey. This follow-up questionnaire was to be used to try to measure the quality of the disability data provided on the Census Test form.

The Census Test was conducted in Melbourne on 24 November 1993, with the follow-up information being gathered when the Census Test form was collected.

The sample of Census Collection Districts (CDs) was selected using estimates made by Statistical Information Services Division (SISD) which would target areas which might contain relatively high proportions of people with severe handicaps. The particular CDs were picked using information derived from the work done on producing synthetic estimates of disability and using the high correlation of age with disability and handicap.

The areas chosen for the test were CDs for which 10 per cent or more of the population were predicted to have severe handicaps. These predictions were based on a simple model using the age and sex characteristics of the CD at the 1991 Census and levels of severe handicap found in the 1988 Survey of Disabled and Aged Persons.

OVERVIEW OF THE RESULTS

There were 29 different CDs selected from various locations in Melbourne for inclusion in the Test, resulting in 2,316 dwellings. After sample loss due to unoccupied dwellings, refusals and interviewer non-contact, 1,644 households were provided with a Census Test form and were subsequently interviewed using the follow-up questionnaire. This sample loss (29%) compares with about 20 per cent in recent 'normal' Census Tests. Details of the sample loss are provided in Appendix 1, Table A3. The higher rate was expected, as several CDs were in coastal areas where there were large numbers of holiday residences. In a few cases, persons had refused because they were too sick or because they had just come out of hospital or were about to enter hospital. It is highly likely that some of these persons would have been coded as having disabilities and handicaps. There were a total of 2,576 persons involved in the test.

Almost 4 per cent of the sample gave incomplete answers to the Census Test questions or did not answer the questions at all (see Appendix 2, Table A5). All the follow-up questionnaires were fully completed for the disability and handicap questions. In a few cases, a Census Test form was completed but a follow-up questionnaire was not, as the interviewer was not able to personally contact the respondent and the Census Test form was left with neighbours or at the door. These cases were treated as non-contacts and included with the non-responses.

As areas of Melbourne had been targeted which were expected to produce a high prevalence of severe handicap, the population interviewed tended to be elderly (50% aged 60 years and over) and in over half the cases (64%) they were living alone (see Appendix 1, Tables A1 and A2, for demographic details of the sample).

The information supplied on the Census Test form about disability and handicap was compared with that obtained in the follow-up questionnaire using the same definitions and techniques as in SDAC. A person was identified as having a disability in the follow-up questionnaire if they gave a positive response to any of the questions concerning impairments, restrictions or disabilities which they might have. They were further categorised as having a severe handicap if they answered that they needed help from another person for the purposes of personal care, mobility or communication (see Disability, Ageing and Carers 1993 User's Guide - Catalogue No. 4431.0 for more details of concepts and definitions).

Table 1 shows the results that were obtained from the test comparisons (see Appendix 2, Table A4, for a more detailed version of the results).

The proportions of false negatives (those categorised as having a severe handicap in the follow-up survey but not in the Census Test) and false positives (those categorised as having a severe handicap in the Census Test but not in the follow-up survey) were substantial. Indeed, over 52 per cent were false negatives. A further 10 per cent (17) of the persons categorised as having a severe handicap by the follow-up survey questions did not provide a codable answer for the Census Test questions on disability. In addition, 31 people (31% of those who would be classified as severely handicapped in the Census Test) did not have this information confirmed in the follow-up survey (false positives). This means that only 38 per cent of those identified with a severe handicap (according to the survey questions) also identified themselves in the same way on the Census Test form.

At an even more detailed level, about two-fifths of the 38 per cent who correctly identified themselves in the Census Test forms, gave the same answer in the Census Test and the follow-up survey about the type of handicap for which their help was needed. There was little difference in the accuracy for the three types of handicaps (eg personal care, mobility and communication).

As can be seen from Table 2 below, there were enormous variations between CDs in the level of accuracy obtained from the answers to the disability questions, as well as in the rate of severe handicap. While the rate of severe handicap is partly dependant on the age structure of the CD, it is obvious that there is little consistency in either the level of handicap or the ability to report that handicap. It is therefore not possible to assume a constant level of under-reporting across all CDs.

TABLE 2: Details of severe handicap at the CD level

Number Percentage with same results in Rate of severe handicapCD identification of persons Census Test and follow-up survey (per 1,000 )

There are few obvious reasons that can be advanced to explain why some people answered the Census Test disability questions correctly and some did not (see Appendix 4, Table A7, for profiles of those who answered the survey and the Census Test questions positively for ove.

Of the people who answered incorrectly in the Census Test, a slightly greater proportion were aged 80 years or more and a slightly greater proportion were female. In addition, those giving incorrect answers identified less impairments and disabilities in the follow-up questions than those who answered correctly. The median number of conditions identified by those giving incorrect answers was 3.0 conditions compared with a median of 4.5 conditions identified by those who answered correctly. Those with speech problems, loss of sight, mental illness or who were slow at learning or who had suffered a stroke were less likely to give incorrect answers than those identifying other disabilities. The wording of the first disability question may have been a trigger for alerting some people with certain types of disabilities to the fact that they should be included, but could have also led to other groups (such as those with arthritis) excluding themselves. It may also be that these types of disabilities very clearly require help from others for the person to be able to cope with the 'activities of daily living'. However, subsequent analysis indicates that the first question only accounts for a small proportion of the error evident in the estimation of people with severe/profound handicap.

The condition which people identified as the reason for needing help probably gives the best explanation for the reasoning behind the answers given in the Census Test. Among those who mentioned arthritis as a condition, 78 per cent answered incorrectly and 75 per cent of those with hearing loss also marked the questions incorrectly. It seems that these types of conditions, which are very much a part of old age, are not recognised as, or admitted to be, disabilities by older people. Therefore, they answered 'no' to the first question on disability and presumably followed the instruction to skip the questions about needing help.

Many people disregard instructions to skip questions so that it might be expected that some would have answered the second disability question after answering 'no' to the first one. While large numbers of people answered 'no' to both questions on disability in the Census Test, only 11 answered 'no' to the first question and 'yes' to the second, and of these only 2 answered positively in the follow-up survey. Some of the respondents who in fact answered 'yes' to the first Census Test question subsequently answered that they did not need help. However, from comments made by the interviewers, as a result of conversations with the respondents, they were clearly receiving help, either from family members or from community service programs.

If the first question had been worded as originally suggested with just 'long-term physical or mental condition' included, it is possible that more of these people with arthritis and hearing loss may have answered correctly. On the other hand, many of the interviewers' reports supported the conclusion that respondents were underestimating both their conditions and their need for help by rationalising that their condition was only part of ageing, so that whatever wording was used they would have been unlikely to answer positively.

For the group of persons who were considered to be false negatives, an estimate was made of the maximum number who might have answered the second Census Test question on disability correctly if they had not by-passed it by answering 'no' to the first question. Applying the proportions of 'yes' and 'no' categories found for the second question where people had answered 'yes' to the first question, suggests that there would be a maximum of 40 additional persons who might have answered correctly. This means that an estimated maximum of 61 per cent of the total persons identified in the follow-up survey as having a severe handicap, might have been expected to answer the second Census Test question on disability correctly. This estimate essentially gives the proportion who might answer this second question correctly as an independent measure, so that the quality of response would not be dependent on the wording of the first question or the order in which the questions were asked (eg there would be no further effect from reversing the order of the questions).

The false negatives

Some positive reasons that could be identified as contributing to the difference in answers given for the Census Test and the follow-up questions by those who were false negatives were (see Table 3):

- Problems with understanding or answering the disability questions in the Census Test due to their complexity;

- Little or no understanding of English;

- Different persons answering the questions in the Census Test and the follow-up survey; and

- Visual impairment.

From the examination of answers to the two sets of questions, these reasons would possibly explain why 45 persons (out of the 93) answered differently for the two questions (some had more than one of the above reasons). However, some of those who answered correctly also had these problems, but to a lesser extent (except for visual impairment), so that these identified reasons may not necessarily have been the contributing factors in producing the differing answers.

It may also be significant that for some cases, when the same person answered the follow-up questionnaire and the Census Test, they managed to answer correctly for other persons in the household but sometimes underestimated or ignored their own needs for help when answering the Census Test question.

Difficulty with the disability questions in the Census Test (from the evaluation question in the follow-up survey) 10 4Little or no understanding of English 10 5Different persons answering the Census Test and survey questions 18 3Visual impairment 15 19

Similar proportions of each group (the correct and the false negatives) were Australian born and similar proportions had left school aged 15 years or less. There was little other data available to explain the differences in responses. Questions were included in the follow-up survey to try to determine the reaction of respondents to the disability questions as far as their ability to understand and answer them. However, due to lack of prior testing, these questions did not produce reliable data which could be used to make an assessment of the questions.

From comments made by respondents (written on Census Test forms) and from comments made by the interviewers it was clear that some of the wording in the questions was causing further problems, mainly due to the inability of people to read the questions properly. There were a few instances of people from non-English speaking backgrounds who answered that they needed help with communicating when there were no disabilities or impairments which might have caused this problem. It seems likely that they failed to read or understand that the problem only related to communicating in their own language. There were even more instances where people had marked that they needed help with moving around and then underlined 'shopping' or had written 'daughter does shopping'. This help was often provided more because of the age of the respondent or their lack of ability to carry the shopping rather than any actual need for help in walking or moving around. In this case, the respondents had focused on the word 'shopping' rather than reading the context of the whole sentence. Probably 'walking to the shops' or 'walking around shops' would have been clearer to understand, but too lengthy for the Census Test form. The measurement of severe handicap is concerned with only those people who need help with the 'activities of daily living' such as taking care of personal needs, moving around and communicating with other people. Needing help to do shopping because of inability to carry heavy loads or just because of age does not place a person in the category of severe handicap.

The false positives

There were 31 respondents who gave a positive response in the Census Test for needing help, but in the follow-up survey, either did not mention that they needed help or stated that the need for help was not due to their disability. There were only four persons for which there was no obvious reason for the differences in their answers, but for the remaining 27 persons there were a number of reasons why they had answered inconsistently, as detailed in Table 4 below. Some respondents were identified as having several of these reasons.

TABLE 4: Reasons for inconsistent answers

Reason for differing answers Number with problems

Visual impairment 7Difficulty but not coded in follow-up 8Help not due to disability 7Problems understanding disability questions 2Different persons answered 7Follow-up questionnaire incomplete or could not be fully understood due to disabilities 5Language problems 9

Where a person had identified that their need for help was not due to their disability, there was little useful information to identify why they were receiving help. However, due to comments about old age and 'help from the council', it is likely that all these cases probably felt that their help was received due to old age, as 5 out of 7 were aged 80 years or more.

COMPARISON OF THE FOLLOW-UP SURVEY AND THE 1993 SURVEY OF DISABILITY, AGING AND CARERS

Disability and handicap rates

The main purpose of this test was to find out how well the answers to the disability questions in the Census could replicate results obtained using methodology similar to that used in the 1993 Survey of Disability, Aging and Carers (SDAC). This has been discussed at length above. In addition, it is possible to test the follow-up results to see how well they compare with SDAC.

To investigate these comparisons, age specific disability rates were calculated for the categories of severe handicap and disability for both the follow-up survey and the household component of SDAC.

Total Rate 36.9 69.4 183.2 385.3 *(36.2) *(235.8)* Rate standardised to the Australian population at March 1993 (eg the SDAC population bench-mark)HH=Household component of Survey only.

As can be seen from the Table 5 above, there were a number of differences in the rates for the two surveys, but no obvious reasons for these differences. Providing the results from SDAC give a correct indication of the level of severity of handicap, it would appear that for severe handicap, those aged 60-69 years were overcounted in the follow-up survey, while those aged 70 years or more were undercounted. This undercounting for the oldest age groups is supported by the anecdotal evidence from the interviewers' reports which often referred to the fact that respondents were underestimating their conditions and need for help. However, the age standardised rate for severe handicap for the follow-up survey was very close to the SDAC rate.

A number of the age specific disability rates are higher in the follow-up survey than in SDAC, but mainly at the younger age groups. The age standardised disability rate for the follow-up survey was also considerably higher than the SDAC disability rate.

There are a number of reasons why the rates found in the two surveys are different:

- SDAC may in fact be missing some people with certain disabilities;

- Persons in the follow-up survey may have overstated their disabilities when questioned. There is no definition or description of what 'restriction' means, so that the interpretation of this may be subjective; and

- The CDs included may not be particularly representative of the total disability found in Victoria or for Australia.

If the follow-up survey had displayed exactly the same rates as SDAC, then it is estimated that there should have been 204 persons with a severe handicap and 841 persons with a disability, compared with the 178 and 988 persons respectively who were actually identified in the survey. The estimated 841 persons with a disability is still considerably higher than the number of persons with a disability identified in the Census Test. There were only 284 persons identified with a disability by the Census Test, of which 261 were confirmed as correct by the follow-up survey. Most importantly, the actual false negative rate would have been much greater than 52 per cent.

Comparison of disabilities and impairments

The proportion of disabilities and impairments identified by the follow-up survey were also compared with those found in SDAC (see Appendix 3, Table A6).

For six types of disability/impairment (out of the fifteen types examined) the proportions for the follow-up survey were at about the level expected. However, for some disabilities (particularly those which result from conditions such as arthritis, which can lead to difficulties in gripping and incomplete use of feet and legs), the follow-up survey showed higher proportions than expected. There were also a considerable number of persons with broken hips and legs, who were expected to be restricted by these conditions for more than six months and who came into the disability category of 'incomplete use of feet and legs', thus adding to the proportion in this category. Proportions of those on medication for long-term conditions, who were restricted by these conditions, and those with other conditions not included in previous questions (eg Alzheimer's Disease, dementia and heart problems etc) were also higher, while the proportion with long-term effects from brain injury, head injury and stroke was much lower in the follow-up survey than in SDAC.

USE OF SYNTHETIC ESTIMATES TO PREDICT DISABILITY

The predictions that the selected CDs would provide a level of severe handicap of 10 per cent (based on data from 1988) prove to have over-estimated the number of people found in this handicap category. From the Census Test, only 178 of the 2,576 respondents (6.9%) proved to have a severe handicap. If the Census Test had found the same age-specific disability rates as SDAC (giving a total of 204 people with a severe handicap) this would have represented 7.9 per cent of the respondents interviewed; still less than the predicted proportion. This may not be surprising as it is likely that, since these people interviewed are mainly living independently, they are more healthy than other people of their age. Thus, while the areas chosen had much higher numbers with severe handicaps than the general population, the proportion was lower than had been predicted, based on age and sex alone.

The ABS is investigating more sophisticated models for prediction of disability status for small geographic areas. The Census Disability Test results will provide a valuable opportunity to evaluate synthetic estimates based on these models.

CONCLUSIONS AND RECOMMENDATIONS

Many problems have been identified when data on disability is collected by self-enumeration. The concept of disability is a complex and subjective one which does not easily lend itself to providing good quality data via self-enumeration methods, such as a Census, where detailed explanations of terms cannot be provided and there is no interviewer to provide additional information. While data on disability collected in surveys through a personal interview seems to be both of a reasonable quality and consistent over time, it is unlikely that Census data would provide either good quality data or a consistent time series.

This test of collecting disability data using Census methodology showed that, even for those with a severe handicap, it is difficult to accurately measure this concept. Based on analysis of this Test, the quality of the data which would be provided via the Census would not be sufficiently accurate to provide usable small area data or data for small population groups. Similar results and conclusions have been found as a result of analysis carried out overseas on the quality of Census data. It is therefore recommended that a question on disability should not be included in the 1996 Census.

(a) No data was collected for children aged under 5 years because it is difficult to determine whether their need for help is due to their disability or just their stage of development.(b) Includes 'not stated' for sex.