Sunday, November 12, 2006

In parts one and two I looked at some of the claims made in Sabrina Freeman's book, "Science for Sale in the Autism Wars", in the section called "The Classic Aversives Ruse".

Another of her claims is that behaviour analysts didn't stop systematically using physical aversives in early ABA programs because this practice became unacceptable. Instead, behaviour analysts found that aversives weren't genuinely effective, and altered their practices accordingly. To support this view, Dr Freeman provides evidence from a 1996 Canadian TV documentary in which Ivar Lovaas appeared and explained the methodological reasons why he abandoned aversives. She also provides a quote from Smith and Lovaas (1997), and also lists three "replication studies" in which aversives weren't used, but which had "significant and impressive" results.

Is Dr Freeman right?

Re Dr Lovaas' turn on TV, using a TV program as a credible source for scientific information seems like a bad idea. But it's consistent with the standards that our finest autism advocates have decided are good for autistics. Autism advocates in Canada have also used education numbers as epidemiology. They've promoted the use of legal decisions, opinion polls, "fact" sheets, political speeches and media reports as credible sources of information about autism and autism treatment, often without bothering with primary sources in peer-reviewed science (see an example here).

Setting aside Dr Lovaas' TV appearance, Smith and Lovaas (1997) is a published rejoinder to criticisms of the UCLA Young Autism Project. It doesn't contain data reporting the success of a non-aversive early ABA program. Instead, in responding to concerns that Lovaas (1987) depended on the use of physical aversives, Smith and Lovaas state that since the time that study began (in the early 1970s), advances in research have "rendered such procedures unnecessary". However, this paper provides no evidence, and no references, indicating what these advances might may be, when they were introduced, when they replaced aversives at UCLA, etc.

As for the three "replication studies" cited by Dr Freeman, one is an uncontrolled trial (Anderson et al., 1987), one is a retrospective study (not a true experimental design) which has been roundly condemned by Dr Lovaas (Sheinkopf & Seigel, 1998; Lovas, 2002), and one is a non-randomized controlled trial which now has a presented follow-up showing that the few autistic children who did well did not maintain this ten years later (Birnbrauer & Leach, 1993, 2006). Not one of these three studies reports even one child who achieved normal functioning in the short term, by the criteria in Lovaas (1987).

So what other evidence is there, besides Dr Freeman's, about why and when aversives stopped being used at UCLA in ABA programs for autistic kids? Is there any evidence for when those advances in research, the ones making aversives obsolete, actually occurred?

Lovaas (2002) writes:

Contingent aversives were employed in the Lovaas (1987) study. Many states now prohibit the use of aversives, and many parents object to such treatment. Either of these factors could prevent replication of the Lovaas (1987) study. However, although the UCLA Young Autism Project no longer uses aversives, we have taken advantage of alternatives to aversives developed during and after the time of the 1987 study was completed (treatment in the Lovaas, 1987, study took place between 1970 and 1984).

Again, there are no references pointing to primary sources reporting data from non-aversive early ABA programs. However, Dr Lovaas doesn't appear to be claiming, as Dr Freeman does, that the unacceptability of aversives (including their illegality) played no role at all in altering the practices of behaviour analysts.

Then there's Smith, Groen and Wynn (2000, 2001). This is the only existing randomized controlled trial of an early ABA program. This study took place at UCLA, involved researchers trained and overseen by Dr Lovaas, and involved, in sequence, all children who were referred to the UCLA YAP between 1989 and 1992. The authors report how their treatment differed from the treatment in Lovaas (1987):

Finally, though contingent aversives were employed briefly with the first 4 children, they were then stopped for all children, whereas Lovaas (1987) employed this intervention more extensively.

So we now know when aversives stopped being used at UCLA: sometime between the 4th and 5th referred child in Smith et al. (2000). It seems unlikely that the original plan was to use aversives with some children and not others; there is, in any case, no mention of this by the authors, who report other abandoned plans. Instead, it seems more plausible that the authors had started by using aversives with the intention of having this as part of the treatment for all the experimental group children. Something then happened between child 4 and child 5 such that child 5 (and all the children after) didn't receive aversives at all, and aversives were removed from the treatment of the first 4 children.

This seems to narrow down the time when, apparently, there were these reported advances in research, and it was decided that they should supplant the use of the now unnecessary and ineffective aversives. Or possibly something else happened. In 1991, a law was passed in California, making the use of physical punishment in behaviour programs illegal. This law was first introduced in 1990. In 1989, it may have already been known that such a law was going to be proposed.

So we know that aversives were considered "effective" at UCLA at least into 1989. If they were not, it would have been grossly unethical to use them on the first 4 children in Smith et al. (2000). It's possible that sometime between child 4 and child 5 in Smith et al. (2000), there was suddenly an advance in research allowing for the abandonment of the suddenly non-effective and unnecessary aversives (for which these researchers must have received the requisite institutional review board permission). Or it's possible that this was about the time it became known that this procedure--physical punishment--was likely to become illegal in California.

Dr Freeman's book was published in 2003, but she doesn't mention Smith et al. (2000), the only direct evidence of exactly when aversives were abandoned by the UCLA YAP. As to why, there is no way precisely to know, unless Drs Lovaas or Smith decide to disclose this information. However, the evidence doesn't entirely support Dr Freeman's account. There's still the problem of Dr Lovaas taking the trouble to prove how effective contingent aversives were in his famous 1987 paper. And the available evidence does strain the credibility of the assertion that it was advances in research which solely were responsible for the UCLA YAP's apparently sudden about-face re aversives.

Good thoughts... I did not read the Sallows paper (2005) as a study with a control group per se nor did I draw the same conclusions regarding it that you have other then we both agree that it does it use a "control group" (as both groups received comparable treatments IMO). It strikes me as more of a pre.-post. design comparring outcomes with children who were receiving ABA-Lovaas in a "home" environment against a "clinic" environment... Both groups were receiveing ABA-Lovaas intervention in different environments. Both recieved intensive intervention with well trained therapists. The home based programs received 6 hours less of 1-1 therapy per week but over 30 hours per week as comparred to 37hours for the clinic group. Both clinic and parent environments seemed to lead to modest +outcomes on the measures used. The authors indicate that data was combined (as there were no significant differences at pre and post measures) as a means of avoiding a confound generated from "poor matching" (my term) of "subjects" in both groups (it was shown that a subset of rapid learners in the "clinc group" had higher pretreatment scores on three measures) . In essence I think that Sallows has shown that you don't need to run ABA-Lovaas in a University center... and that positive outcomes can be the result when ABA-Lovaas is used in home environments... but, due to the lack of a true control group... that is about all you can say.

Sallows and Graupner (2005) isn't a study of Lovaas-type ABA. The authors' description of their treatment includes non-Lovaas behaviour analytic approaches (e.g., from the Koegels' 1995 book) and non-behaviour-analytic approaches (e.g., social stories).

An absence of correlation between intensity of treatment and outcome measures was found, regardless of which group the participants were in. As I've written elsewhere, this uncontrolled trial demonstrates that neither intensity nor quality of ABA/IBI is relevant to outcomes.

Heh... yep... not 100% Lovaas... but as they say "Lovaas does not do Lovaas any more...". Perhaps evidence that things have infact moved on and aversives are in fact no longer needed (as per Lovaas' "explanation" re discontinuing aversives mid project...).

I think your point re "intensity nor quality of ABA/IBI is relevant to outcomes" 'is over stating the case. It was shown that no statistically significant differences were found between well run "30 hour" home programs and "37 hour" clinic based programs... extrapolation from these results to conclude that a 14 hour program would have the same impact as a 40 hour program is overextending the evidence IMO. In this case 30 hour home programs produced similar results as a 40 hour clinic program. I wonder if the clinic environement had variables that served as confounds (over supervision/micro management of therapists, lack of spontaneoius play, unfamilair or less favored learning items....).

Michelle... you continue to challenge my assumptions which is a good thing. Thanks.

Again, I'm not comparing the original two groups (experimental and control). What I mean is a correlation between hrs/wk and outcome measures (this is stuff researchers do--we make scatterplots). And there was none. This is only vaguely stated in the paper, so I contacted Dr Sallows and asked. And no, there was no correlation between treatment intensity and outcome measures. So I'm describing what was found, which is not "overstating the case".

Dr Lovaas is scheduled to present (with two colleagues) a paper later this year, on the subject of how and when to use aversive and restrictive procedures in behaviour interventions.

There are also two recent Richard Foxx papers which involve autistics and aversive procedures (overcorrection, restraints).

I have read some of Foxx's work... and I will await Dr Lovaas' paper but I must admit that I am very concerned that this remains part of the "conversation" within the ABA community. I would imagine that some form of arguement will be presented that will be based on "cost-benefit" matrix (outcomes of treatment with aversives out "weigh" the risks of not using them) with some caveat that "eveything else" must be tried first. I am sceptical that I will be moved to accept the argument presented... maybe it would be better then "chemical" interventions? I doubt it. I suppose that wen a client has truely self destructive behavior or is super aggressive... nah... I don't think I can see my self acepting it. I will wait to see what is written.

So... no correlation between hrs/wk and outcome measures as reported in this study... is this due to the rough equivelence of both treatment protocols? Both treatment groups had skilled interventionists, consultants, and hrs/wk (6 hours variance)... I suspect that such a relationship would be shown if a group had been set up to recieve 14 hours per week... but there was not. Clearly I am missing the point. Sorry. I'll dig up my Psych Stats text again and see if I can suss it out.

The absence of correlation between treatment intensity and hrs/wk is the result of... there being no relationship between a quite wide range (setting aside the 14hrs/wk outlier) of hrs/wk (a range of at least 20-40hrs/wk in the combined first two years) and the outcome measures used in this study.

The authors helpfully show how it's done, by dismissing the possibility that "supplemental treatments" (received by all but one child) had any effect on outcomes. How did the authors do this? They used correlations between intensity of these additional treatments and outcome measures. They found no significant correlation. So they concluded that this lack of correlation means that "the increases in skills observed in this study were not the result of these interventions."

Dr Lovaas and colleagues are presenting their paper about aversives at a conference this spring (in case the fact that this was a presentation was not clear).

I disagree that the two original groups in Sallows and Graupner (2005) received equivalent treatments; I can imagine the trouble I would get into if I suggested (for starters) that ABA consultants contribute much less to child outcomes than is currently assumed to be the case.

... opps forgot to add to the last comment... within the paper the authors do note that "...Because the parent-directed children as a group received.. only 6 to 7 hours less then the clinic directed group, our ability to examine the effect of differneces in treatment intensity is limited"... I guess they are suggesting/asserting that treatment intensity would be difficult to to isolate as a variable as the two treatments (intensisty) were close.

Again, when I state that this study shows that intensity of treatment is irrelevant to outcomes, this is not referring to any difference between average hrs/wk per group in the original two groups (experimental and control). The original groupings (into experimental and control conditions according to quality of treatment) had no effect on outcome measures.

I'm referring to whether there was or wasn't a correlation between hrs/wk and any of the chosen outcome measures. And there wasn't.

In order to show that treatment intensity has an effect (presumably, you're looking for a positive effect) on outcomes measures, there should be a significant association between the two. Through the wonders of statistics, this is something that can be found out. You might want to look this up yourself, when you have some spare time--find out what scatterplots are, what r means, and so on.

...in order for treatment intensity to show an association on outcome measures would you not need two distinct and significantly different degrees of intensity of treatment for such a correlation to be seen clearly?