In the previous chapter we found that attrition produced no systematic pattern of treatment/control differences in the observed, initial characteristics of the samples available for analysis. This finding does not rule out the possibility that at the time of follow up the two groups differ on unmeasured characteristics that also affect the level of a particular outcome measure. If, as a consequence of attrition, the treatment and control groups differ on average on these unmeasured characteristics, then impact estimates that do not control for these differences will be biased.

In this chapter we outline, first informally and later using statistical notation, the conditions under which attrition bias may arise. Next, we outline a statistical procedure, due to Heckman (1976, 1979) that corrects for possible bias. Finally, we show how the direction of the bias can be determined when some prior knowledge exists concerning the mechanism causing attrition.

We focus here on giving a heuristic explanation of the procedures. More complete coverage of statistical details and derivations of these methods and their justification is given in the references listed at the end of this report.

As indicated in the previous chapter, differences between treatment and control groups on screen characteristics that arise because of different patterns of attrition do not necessarily imply that estimates of channeling impacts are biased. If nonresponse was affected only by screen characteristics (e.g., ADL), then inclusion of these screen characteristics as explanatory (auxiliary control) variables in the outcome regression would control for the effects of attrition. This fact is not widely understood. Two conditions are necessary for impact estimates to be biased: (1) attrition is affected by the outcome measure being examined (e.g., whether in a nursing home), or by some unobserved factor (e.g., health status at the time of the attempted followup) that affects both attrition and the outcome measure, and (2) the pattern of attrition differs for treatment and control groups. Thus, finding attrition-induced differences between treatment and control groups on observed characteristics would have implied two things. First, impacts would have to be estimated by regression controlling for all initial characteristics on which treatments and controls differ as a consequence of attrition. Second, attrition-induced differences on observed characteristics would raise the suspicion that treatments and controls differ on unmeasured characteristics as well.

To see how different attrition patterns affect impact estimates, consider the following example. Suppose that in the full sample, 20 percent of the treatment group and 30 percent of the control group are in a nursing home at the 6-month followup. Thus, channeling reduced the probability of being in a nursing home by 10 percentage points. However, contrast the effects on these results under two different assumptions about the mechanism governing attrition. In the first case, assume that attrition was random within each experimental group--that is, attrition was affected only by experimental status: controls have a 70 percent probability of response and treatments have an 80 percent probability. Clearly, restricting the analysis to just the responders would have no effect on the impact estimate: since all treatment group members have the same probability of response, the proportion of responders in nursing homes is the same as the proportion for the full sample (20 percent), and similarly 30 percent of the controls who respond to the interview will be in nursing homes. Hence, the impact estimate of 10 percentage points is unaffected by attrition, even though the attrition rates are different for treatments and controls. Furthermore, although we do not show this here, the same results hold if attrition is affected by any screen characteristic that is controlled for.

Consider how our conclusions about the effects of attrition change, however, if the probability of response is also affected by the value of the outcome being examined. For example, suppose that the probability of response among those who are in a nursing home at followup (Y=I) is only 50 percent for treatment group members and 40 percent for control group members. Suppose that the probabilities of response for those not in nursing homes (Y=0) are 95 percent for the treatment group and 85 percent for the control group (i.e., for each value of Y, treatment group members are 10 percentage points more likely to respond than controls). In this case, the treatment and control group means for the responders only (R=1) are:15

Treatment group:

Estimated proportion in nursing homes for responders

=

responders in nursing homes

(responders in nursing homes + responders not in nursing homes)

=

0.20 * 0.50 * N / (0.20 * 0.50 + 0.80 * 0.95) * N

=

11.6%

Control group:

Estimated proportion in nursing homes

=

0.30 * 0.40 * N / (0.30 * 0.40 + 0.70 * 0.84) * N

=

16.8%

Note that the proportion of responders in nursing homes is much smaller than the values that would have been obtained if no attrition occurred (about half as large). More important, however, we see that subtracting the control group mean from the treatment group mean for respondents gives a predicted impact of channeling on nursing home placement of -5.2 percentage points, about half the true impact in this example.

These overly simplified examples demonstrate how different mechanisms of attrition may or may not cause bias in impact estimates. Overall attrition rates for treatment and control groups in the second example are 14 and 28.5 percent, respectively--quite close to the rates actually observed for some of our analysis samples. An attrition mechanism of this type could result in estimated impacts that are too small to be statistically significant, for cases in which the estimate for the full sample would have been large and highly significant.

The statistical correction procedure described in this chapter controls for attrition by determining whether responding sample members who have a relatively low predicted probability of remaining in the sample (given their screen characteristics) are more likely to have a larger (or smaller) than expected value of the outcome (Y), given the values of auxiliary control variables. This is determined by obtaining for each observation a predicted "attrition-correction" term and including it in the regression equation used to estimate channeling impacts. If this type of correlation between unobserved factors in the attrition equation and unobserved factors in the outcome equation does exist: the coefficient on the correction term will have a significant coefficient. If, in addition, attrition is affected by experimental status, the estimated channeling impact will change substantially. The statistical procedure is described below.

Regression analysis was used throughout the evaluation of channeling impacts to eliminate potential bias in impact estimates that could arise due to treatment/control differences on observed characteristics at baseline, to control for the different distribution of treatment and control groups across sites, and to provide more efficient impact estimates than a simple treatment/control group comparison of means would yield. The regression model used to estimate channeling's impacts can be described as follows. Let Y1 be the outcome of interest, such as number of hospital days or number of nursing home days. Define TB = 1 if the sample member belongs to the treatment group in the basic case management model, and TB = 0 if he or she does not. Similarly, define TF = 1 if the sample member belongs to the treatment group in the financial control model, and TF = 0 otherwise. Finally, define a set of auxiliary control variables (X1) such as site, sex, race, income, and impairment in functioning. These variables are included in the outcome equation to control for preexisting differences between sample members on characteristics that affect the value of the outcome. The model is then

(1)

Y1

=

aBTB + aFTF + X1b1 + u1

=

Zb = u1

where aB and aF are the estimates of the impact of channeling on the .outcome, Y1, for the basic and financial control models, respectively; b1 is a vector of coefficients on the auxiliary variables; and u1 is the disturbance term capturing all of the unobserved factors which influence Y1. To facilitate the exposition below, this equation is rewritten in terms of Z and b, where Z is a vector that contains variables TB, TF, and X1, and b is the true, unobserved value of the regression parameters (aB, aF, and b1) in equation (1). In the absence of sample attrition, if the random assignment to treatment or control groups was performed correctly and the usual assumptions of least-squares regression are satisfied,16 then regression estimates of aB and aF are unbiased estimates of the impacts of channeling on outcome Y1 for the basic case management and the financial control models, respectively.

As noted above, however, we are not able to estimate this model on the full sample because of attrition. To the extent that the included auxiliary control variables (X1) account fully for the effect of any differences between responders and nonresponders on the outcome variable (Y1), the estimated coefficients in equation (1), including aB and aF, remain unbiased. However, if there are unmeasured characteristics that affect both the probability of attrition and the outcome of interest, the estimated coefficients in equation (1) will in general be biased.

The following exposition describes the mechanism by which this bias occurs. Suppose that the attrition process can be described by the equations:

(2)

Y2*

=

X2b2 + u2, and

(3)

Y2

=

1 if Y2* > 0 (in the analysis sample)

0 if *Y2< 0 (lost from sample due to attrition).

The dependent variable in equation (2) is an unobserved continuous variable, Y2*, representing the sample member's propensity to respond to the interviews (or otherwise be included in a particular analysis sample). Each sample member has his or her own tendency to cooperate with the research and refuses when the perceived effort to respond exceeds a tolerance. This tendency to respond is not observed directly, but individuals with values exceeding a constant--without loss of generality assumed to be zero--are observed to respond (Y2 = 1), while those with values less than or equal to zero are nonresponders (Y2 = 0). Propensity to respond is assumed to be a function of observable characteristics, X2 (which includes treatment status and may include other variables also included in X1), as well as unobservable characteristics and circumstances, represented by the disturbance term u2, assumed to follow a standard normal distribution.17

Bias arises in the estimates of aB and aF if the unobserved factors affecting attrition (u2) are correlated with the unobserved factors (u1) that affect the outcome measure (Y1). This can be seen by examining the general expression for the vector of regression coefficients for equation (1), which we will refer to as b~:

(4)

b~

=

(Z ' Z) -1Z ' Y1,

=

b + (Z ' Z) -1Z ' u1.

Without sample attrition, the expected value of the estimated regression coefficients is the true value of the parameters (b), because the last term in the expression above has an expected value of zero. With attrition, however, the expected value of b~, given that it is estimated on only observations in the analysis sample is:

(5)

E(b~ | in the analysis sample)

=

b + E [(Z ' Z) Z ' u1 | Y2 = 1]

=

b + E [(Z ' Z) -1Z u1 | Y2* > 0]

=

b + E [(Z ' Z) -1Z u1 | u2 > -X2b2]

=

b + (Z ' Z) -1Z E [u1 | -X2b2].

If u1 and u2 are correlated (i.e., if there are unobserved factors that affect both Y1, and the probability of attrition), the expected value of the final expression in square brackets will not be zero, and therefore the expected value of the regression estimates of the parameters of equation (1), including the expected value of the estimates of aB and aF, will not be equal to the true values of these parameters. Thus, the estimates are biased by sample attrition, and the size and direction of the bias are unknown.18

The nature of this bias and a procedure for correcting it were expounded by Heckman (1976, 1979). Heckman showed that the bias due to sample attrition is analogous to the bias due to omitting an important explanatory variable. That is, we have

(6)

E(Y1 | Y2* > 0)

=

Zb + E(u1 | Y2* > 0)

=

Zb + E(u1 | u2 > -X2b2).

As noted above, one of the assumptions of least squares regression is that the expected value of u1 is zero, so estimates of b will be unbiased. However, when sample attrition exists, the regression can be estimated only on those sample members with complete data, so unbiasedness of the resulting estimates requires that the expected value of u1, conditional upon the sample members' availability for analysis, be equal to zero. If u1 and u2 are correlated, however, this conditional expectation of u1 is not zero but is a function of u2 and X2. In this case, if Y1 is regressed on Z, and there is correlation between the variables in Z and those in X2, regression estimates of b will be biased because an "omitted" term (the nonzero conditional expected value of u1) is correlated with the regressors Z. The estimated coefficients on the variables in Z, including those on treatment status, will reflect not only the effect of Z on Y1, but also the relationship between Z and the conditional expectation of u1.

In this evaluation attrition could lead to bias in estimates of channeling impacts because those conditions that lead to bias may well be present. For example, suppose that the sample members who are the most impaired at followup are least likely to respond and also likely to have systematically higher (or lower) values of Y1 (e.g., hospital days). Since the auxiliary control variables measured at screen or baseline do not fully reflect impairment levels at the time of followup, u1 and u2 will be correlated. Furthermore, many of the variables Z and X2 that affect the outcome and the likelihood of attrition, respectively, are likely to be the same or to be highly correlated (e.g., both the outcome and likelihood of attrition may be affected by treatment/control status). Thus, there is a strong possibility that the two conditions that together produce biased estimates of regression parameters may be present and, therefore, that estimates of channeling impacts will be biased by attrition.

Fortunately, with an additional assumption, a statistical correction for attrition bias is possible. Heckman showed that although the second term on the right-hand side of equation (6) is unobserved, the n term has a relatively simple form if u1 and u2 are assumed to have a bivariate normal distribution, and this term can be estimated. Heckman shows that

(7)

E(u1 | u2 > -X2b2

=

12

f(X2b2 / 2)

2

F(X2b2 / 2)

=

(12 / 2)M

where 12 is the covariance of u1 and u2, 2 is the standard deviation of u2, b2 is the vector of the estimated coefficients from the attrition equation, f(X2b2 / 2) is the standard normal density function evaluated at X2b2 / 2, and F(X2b2 / 2) is the standard normal distribution function evaluated at the same point. If the parameters b2 of the attrition equation were known, the term M could be constructed for each sample member and used as an additional variable in the regression model. Inclusion of this variable in this regression eliminates it from the error term and therefore eliminates the correlation between Z and the error term in equation (6), thereby eliminating the (asymptotic) attrition bias in estimates of b. The regression coefficient obtained on this M term is an estimate of 12 / 2, the (normalized) covariance between u1 and u2.

The parameters b2 are not known, but can be readily estimated. Thus, the procedure developed by Heckman and used in this report to eliminate attrition bias can be described as follows:

Using all observations (both responders and nonresponders), estimate the parameters of the attrition model given in equations (2) and (3) using maximum likelihood probit.19

From the estimated probit coefficients (b2) and the data on X2, form the correction term (M) for the observations which have valid data for the outcome regression--this excludes those lost due to attrition--and estimate equation (8) by least squares:20

(8)

Y1

=

aBTB + aFTF + X1b1 + cM = u1*,

where this equation is simply equation (1) with the nonzero conditional expectation (cM) of the old disturbance term plus a new disturbance term (u1*) substituted for the old disturbance term (u1). The statistical significance of c, the coefficient on M, is an indication of whether there are unobserved factors affecting both attrition and Y1, a necessary condition for the estimates of aB and aF to be biased.

In the discussion of results in the next section, we assess the extent of attrition bias in estimates of channeling impacts in two ways: first, by examining the estimate of c, to determine whether the condition necessary for bias is met, and if so, the size and sign of the correlation between the two disturbance terms; and second, by comparing the regression estimates of aB and aF obtained when potential attrition bias is not controlled for (i.e., from estimating equation (1)) to the impact estimates obtained when this potential bias is controlled for (by estimating equation (8)). In interpreting these results, it is useful to bear in mind the determinants of the bias in a particular coefficient. Inserting the expression in equation (7) into equation (5), the bias in the uncorrected estimates of aB, aF, and b1 is shown to be

(9)

bias ≡ E(b~)-b

=

(12 / 2) (Z Z) -1Z M

=

(12 / 2) PZ,M,

where the term PZ,M is a vector of auxiliary regression coefficients obtained from regressing the constructed M term on the other variables (Z's) in equation (8).21 Thus, the bias in the regression coefficient on any particular explanatory variable (e.g., aB, the coefficient on TB in the outcome equation) is equal to the covariance between u1 and u2 (normalized by 2), multiplied by the coefficient on this same variable (e.g., TB) from a second, auxiliary regression of the constructed M variable on all of the Z variables.

The usefulness of this expression is best demonstrated by elaborating on our previous example. Suppose that we are interested in estimating the impacts of channeling on the number of hospital days (using a followup sample). Also, suppose that those who are most impaired at the time of the followup are less likely to be available for analysis than are less impaired sample members and that the effects of this impairment on hospital days is imperfectly controlled for with the baseline control variables. Since the most impaired individuals are most likely to be in a hospital and least likely to be in the analysis sample, the covariance between u1 and u2 (12) will be negative. Furthermore, since treatment group members are more likely to be available for analysis than control group members, it can be shown that the auxiliary regression coefficient of treatment status contained in PZ,M is expected to be negative.22 Thus, we would expect the attrition bias in the estimate of aB to be positive. That is, the estimated impact will be a larger number than it should be. Thus, we could find an estimated impact of zero when in fact the impact was negative, implying a reduction in hospital days due to channeling. This analytic assessment of the direction of bias is consistent with the heuristic argument that the sample members most likely to be lost to analysis are control group members with relatively large numbers of hospital days, and if these cases were appropriately represented in the analysis sample, the treatment/control difference in expected hospital days would have been a larger negative number. Based on this reasoning, the following reference table can be used to draw inferences about the expected direction of the bias (if any) due to attrition in estimates of channeling impacts:

Expected Relationship (12) Between Outcome (Y1) and the Likelihood that Sample Member is Available For Analysis

Using equation (9), knowledge of the sign of 12 is sufficient to determine the expected direction of the bias. This results from the fact that we know that response rates are higher for the treatment group, and therefore, that the coefficients on TB and TF in the auxiliary regression of M on TB, TF, and X2 are negative.

This interpretation may be confusing for impacts that are expected to be negative in sign, such as impacts on hospital or nursing home use. For example, if the bias is expected to be positive, this means the estimate is too large, i.e., it should be a larger negative number (e.g., -6) rather than a smaller negative number (e.g., -2). Thus, the reduction due to channeling is understated. On the other hand, if the estimated impact is expected to be positive in sign, such as impacts on case management, and the bias is positive, the impact of channeling is overstated because of attrition bias. Thus, the expected direction of channeling impacts as well as the expected direction of the bias are required in order to draw the desired inferences about the effects of attrition on channeling impacts.

Using this table for our example, we expect 12 to be negative because those who are most impaired are likely to have more hospital days, but are less likely to respond. Thus, the expected bias in the impact estimate is positive, and since channeling is predicted to reduce hospital days, the estimated reduction in hospital days will be understated if attrition bias is not corrected for.

Survey Disclaimer

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0379. The time required to complete this information collection is estimated to average 5 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer.