Experimentation For Pretesting Public Health Programs: the Case of the Anti-Drug Abuse Campaigns

Michael L. Ray, Stanford UniversityScott Ward, Harvard University

ABSTRACT - In recent years, much attention has been devoted to "broadening the marketing concept," i.e., applying marketing concepts and techniques to problems beyond the traditional private enterprise sector. One such zone of application has been public health related problems. As health-related fields realized the necessity of effective communication with target audiences, social researchers in universities and consulting agencies were quick to respond to the resulting need for campaign evaluation research. Comparatively little has been done to illustrate to public health professionals the application of marketing concepts and measures to problems involved in pretesting. By "pretesting," we are referring to the activities of defining communication objectives, generating message alternatives, selecting the most effective messages and campaign strategies, and evaluating results.

EXPERIMENTATION FOR PRETESTING PUBLIC HEALTH PROGRAMS: THE CASE OF THE ANTI-DRUG ABUSE CAMPAIGNS

Michael L. Ray, Stanford University

Scott Ward, Harvard University

[This research was supported by the National Institute of Mental Health (NIMH-OC-72-156) and the Marketing Science Institute.]

ABSTRACT -

In recent years, much attention has been devoted to "broadening the marketing concept," i.e., applying marketing concepts and techniques to problems beyond the traditional private enterprise sector. One such zone of application has been public health related problems. As health-related fields realized the necessity of effective communication with target audiences, social researchers in universities and consulting agencies were quick to respond to the resulting need for campaign evaluation research. Comparatively little has been done to illustrate to public health professionals the application of marketing concepts and measures to problems involved in pretesting. By "pretesting," we are referring to the activities of defining communication objectives, generating message alternatives, selecting the most effective messages and campaign strategies, and evaluating results.

This paper summarizes a research project which sought to develop pretesting methods for federally-funded anti-drug abuse media campaigns. The project consisted of two large-scale experiments in laboratory settings, and a field experiment utilizing a split-cable facility. The field experiment was designed to provide field comparison data to tentatively assess the validity of less natural laboratory predictions.

The project focused on selection and scheduling of commercials, i.e., the problem of selecting one finished television commercial among several alternatives designed for various audience segments, rather than the problem of developing procedures for "idea generation,'' or some other aspect of pretesting. The project used finished television commercials, but the procedures developed here could readily be adapted for use with other materials (e.g., storyboards), and the procedures could also be adapted for pretesting advertising for media other than TV.

OVERALL PROJECT OBJECTIVES

In planning and conducting this research, we pursued the following general objectives.

1. To develop a pretesting technique that is reliable, valid, and sensitive to different audience effects produced by various advertisement alternatives.

To this end three different commercials -- "Walkout," "Big Brother," and "People" -- were tested among three different audience groups. Two large-scale laboratory experiments were conducted in order to test and refine measures and procedures, and a field experiment using a split-cable facility was conducted in order to validate these laboratory procedures.

2. To develop a practical pretesting method.

To this end the procedures we recommend could provide preliminary data within two weeks after receipt of test advertising; out-of-pocket costs would be approximately $1,750.

3. To create a test environment which approximates "normal" television viewing, i.e., in which conditions are manipulated to reflect "real world" viewing conditions, and which can be expected on theoretical $rounds to affect audience responses to advertising in important ways.

In order to accomplish this objective, subjects were exposed to the test advertising in naturalistic surroundings, and the ads were imbedded in a 17 1/2 minute segment of programming. Additionally, we varied the level of distraction present during the pretest, and the nature and amount of competing messages during exposure.

4. To employ a range of dependent ("effects") variables which reflect ions-term and short-term cognitive and behavioral effects of advertising and which are consistent with advertising objectives.

To this end, we measured cognitive responses to the advertising which occurred during exposure, by asking subjects to recall what they thought about while viewing. From these questions, we found subjects frequently "counterargued" against the messages, and/or "connected," i.e., experienced a mental link between message content and self-experience. We also examined the following effects:

*

recall of points made, sponsor, situation shown;

*

changes in ranking of the "drug abuse problem";

*

attitudes about anti-drug abuse;

*

interest in receiving a booklet about anti- drug abuse;

*

booklet reading; and

*

stimulation of discussions about anti-drug abuse with family or friends.

OVERVIEW OF PROCEDURES

In pursuing our study objectives, we employed a technique which involved each respondent in the following series of events:

a. Come to a central location

b. Read "Television Violence-Humor Project," the cover story for the experiment.

c. Complete sociodemographic and media exposure questionnaire.

d. View a 172 minute program on a closed-circuit television set. The program contains two exposures of test anti-drug abuse commercials as well as regular programming seen on television.

e. Complete self-administered questionnaires with items relating to the cover story, as well as items eliciting information on recall of advertising, cognitive response, attitudes toward social problems, and interest in receiving relevant booklets. Specific attention is not paid to drugs or anti-drug abuse advertising until the last questionnaire.

f. Sometimes, following the above study procedures, drug booklets are sent to respondent's homes and follow-up telephone interviews are conducted to determine their reactions.

The technique developed in this project can be adapted to different types of respondents (e.g., children vs. adults), different test materials, viewing conditions, and measures employed.

Reliability and validity of the pretesting technique was assessed by examining consistency of results across three of the project studies -- two laboratory studies and one split-cable field experiment. The technique appears to work well because it simulates natural viewing and response processes, a simulation produced by conditions of distraction and attention developed in the first study. It should be pointed out, however, that the reliability and validity of the technique is limited to gross indications of the relative effectiveness of advertisements in various conditions. No pretesting technique can produce accurate absolute predictions, but this technique will offer excellent guidance for public health campaign planners.

The split-cable field experiment showed that it is possible to produce meaningful effects with a heavy anti-drug abuse campaign in a short period of time (one month). A campaign with such a heavy emphasis could only be carried out in short, local efforts, but such a campaign seems well worthwhile given the frequent, local, epidemic character of the drug problem. The campaign would have promise for promoting an atmosphere of belief in which other anti-drug abuse activities could work effectively.

Three sets of dimensions are relevant in evaluating selection or scheduling pretesting procedures:

*

design, measurement, and sampling dimensions;

*

creative (or stimulus) research dimensions; and

*

cost and payout dimensions.

For each set of dimensions, campaign planners must make decisions based on the naturalness or the artificiality of the pretest research setting. A completely natural test would involve post-testing, or running a complete communication campaign on the media and observing responses in terms of actual drug abuse. A completely artificial pretest would be the type referred to above as a developmental pretest, the kind almost always conducted in anti-drug abuse campaign development. Obviously, if there were unlimited funds and time, the most natural type of test would be desirable. It would be totally valid and would provide much valuable information for long-range development. On the other hand, less natural forms of pretesting are less expensive and have great short-term value because they allow for changes in campaigns before running them full-scale with possible damage.

The three types of pretesting discussed to this point can be arrayed on a dimension from natural to artificial, as shown below:

As can be seen, the selection/scheduling type of pretesting is arrayed near the middle of the natural to artificial dimension. Thus, this type of pretesting has the advantages of a relatively high degree of validity and precision which only a completely natural test can achieve, as well as the efficiency of the more artificial types of pretesting.

Design, Measurement and Sampling Dimensions

The specific dimensions involved in design, measurement and sampling considerations are arrayed on the natural to artificial scale in Figure 2.

As the figure illustrates, campaign planners must make decisions concerning research design: should the research be conducted in home, or in an artificial research facility? Measurement and sampling decisions must also be made: should the measure be unobtrusive (i.e., totally natural); should repetition effects be taken into account, should the sample be representative and random, or can an "accidental" sample be tolerated? Perhaps most important, should the design and measurement plan be based on total experimental control (i.e., of temporal order in presentation of test stimuli), or should the research be "one-shot," with no comparison or control possibility?

Creative (or Stimulus) Research Dimensions

A second set of dimensions involves decisions about what, specifically, subjects or respondents will respond to in the pretest. The pretest dimensions involved in subjects' responses to various types of test ads are arrayed on the natural to artificial scale below.

Key decisions concern whether individuals will view complete, finished campaigns, or simply rough cuts or storyboards. Another important decision concerns whether people view test ads in isolation, or in the context of other messages. If respondents simply view a test ad on a tear sheet, this is obviously more artificial than viewing the ad in the context of a dummy magazine. In the case of television messages, it is obviously more "natural" to view test ads via a television received rather than via a movie screen; and it is also more natural to include test messages in the context of programming and other messages, rather than in isolation.

Cost and Payout Dimensions

Finally, the campaign planner must make decisions based on costs, and on how quickly the results must "payout". The pretest costs, time, and payout dimensions involved in planning media campaigns are arrayed on the natural to artificial scale below.

In the first study, conducted in a laboratory setting, tests were run on three 30 second anti-drug abuse ads, ostensibly aimed at three different audience groups:

*

The "Walkout" commercial, aimed primarily at parents of adolescents and preadolescents, was intended to stimulate concern about drug abuse, and interest in receiving more detailed information (a booklet).

*

The "Big Brother" commercial was intended to stimulate older adolescents to warn younger siblings about the danger of drug abuse.

*

Compared with the above, the "People" commercial was more amorphous in its objectives. It contained "face shots" of various individuals (including celebrities) remarking about the drug abuse problem, and concluded with an appeal to the viewer to send for a booklet about it.

For selecting independent variables other than the ads (e.g., viewer attitudes, distractions, or competing messages which resulted in the commercial having greater or lesser impact on viewers), the following criteria were used:

1. Dependent variables should be valid, reliable, sensitive and practical.

2. Multiple measures should be employed to illustrate the range of effects.

3. The measures should reflect people's thoughts while watching.

4. The measures should be consistent with advertising objectives -- namely, to measure both cognitive response and behavioral reactions.

Independent Variables

*

Level of Distraction

A key variable employed in the first study was the level of distraction present during exposure. As television viewing normally occurs under distracting conditions--people talking, etc.--the theoretical literature suggests that such distraction could be expected to affect responses to ad stimuli. In early persuasion research, investigators suggested that subjects generate counter-arguments while attending to messages opposed to their position, and that distraction while subjects attend to persuasive messages might inhibit counterargument production, thereby facilitating attitude change. However, distraction cannot be so great as to eliminate message comprehension, and the topic must be a salient one for subjects (see Festinger and Maccoby, 1964; Baron, Brown, and Miller, 1973; Roberts and Maccoby, 1974).

In the present study, we expected varying degrees of commitment to the position that drug abuse is a serious problem, that it is harmful, etc. Adults should be most committed to the position advocated in the ads, while teenagers and preteenagers may he less committed. In any case, we expected varying levels of distraction during viewing to alter the amount and kinds of thoughts these different audience members experienced during viewing.

*

Competing Messages

A second independent variable employed in the first study was amount and kind of competing messages--i.e., other public service ads, advertising for over-the-counter drug products, and a control condition (no competing ads).

Dependent Variables. In most pretesting research, responses to ads are obtained following exposure. Such viewer reactions may take the form of several cognitive responses, such as counterarguments against the message, the situation portrayed, etc., or their reactions may be in the form of thoughts which essentially link message content and personal experience ("connections"). These responses most often concern reactions at the time the question is asked, and considerable bias can enter into the question/response procedures.

Consequently, we presented an open-ended question, asking respondents to describe "what they thought about" while watching the ads. On the basis of previous research, we expected most reactions to be in the form of counterarguing, and "connections"--which follows from research and theoretical notions of Herbert Krugman (1968a, 19685). Often investigators have posed different cognitive responses in persuasion research, e.g., "rehearsal" of message arguments (Kelman, 1953); counterarguing support argument, source derogation, and curiosity (Wright, 1972).

Based on this research, the objectives of the messages, and our knowledge of the orientations of the subject populations from previous research on drug attitudes and usage (Johnston, 1973; National Commission on Marijuana and Drug Abuse, 1972), we specified the following general expectations for our data.

1. Since adults can be expected to hold attitudes consistent with the anti-drug abuse messages, they should counterargue less, and experience more connections than teenagers. However, since the anti-drug abuse messages contain unpleasant, perhaps threatening information, adults should be particularly sensitive to level of distraction due to the high degree of cognitive effort required to counterargue or connect.

2. Many senior-high-school students can be expected to be opposed to the anti-drug abuse messages, and to possess much information and extensive predispositions about drug abuse. Therefore, they should counterargue more than other groups, but increasing distraction should disrupt counterargument production.

3. For junior-high students, effects should be greatest in the low distraction condition. These younger students should have less information about drug abuse and less developed attitudes than older teenagers and parents. Consequently, they should exhibit low levels of counterarguing and connections, and distraction should dampen messages' effects further.

Research Procedures Used in Study One

The first large-scale experiment was conducted in Palo Alto, California (n=438). Subjects included three population groups: junior-high-school students (n=90), senior-high-school students (n=91), and parents (n=257). Subjects from each group were told that they were participating in a "Television Violence-Humor Project," and their opinions were sought in this area. Subjects viewed a 172 minute videotape of a popular television program, which contained one of the test messages, as well as other (commercial) advertisements. There were three levels of distraction: none, low, and high. Low and high levels were differentiated by decibel levels of a tape recording during the experiment (a male-female conversation).

Following exposure, subjects completed post-exposure questionnaires. They were asked if they remembered any of the commercials, and if so, "what they thought about" while watching. The latter question provided data on cognitive responses--counterarguments and connections--to be reported here. Other questions concerned ranking of drug abuse as a social problem (providing a listing of 15 social problems), and interest in receiving booklets related to all of the commercials embedded in the programming. Two of the booklets were referred to in the anti-drug abuse commercials to which subjects had been exposed.

Effects of Inductions

In persuasion research, level of distraction cannot be so dominant as to greatly reduce message comprehension. In the present research, effects of distraction were gauged through a self-report distraction scale, and through analyses of interactions of ad stimulus and recall with distraction level.

Results showed that the distraction induction was successful. The effect of distraction on recall varied markedly by ad stimulus and subject population, suggesting that distraction did not simply "wash out" opportunities for learning, but produced different results in interaction with other variables.

Cognitive Responses: Counterarguing and Connections

Since small sample sizes preclude reliable analysis of three-way interactions between subject population, distraction, and ad stimulus, data are presented by subject population and ad stimulus (Table 1) and by subject population and distraction (Table 2). In both cases, for those respondents in each group who indicated that they recalled the ads, "percent recalling" refers to the proportion who chose to respond to an open-ended question concerning their thoughts while viewing.

In Table 1 data are shown which are consistent with our expectations. Students chose to comment less about their thoughts while viewing than did parents, and the average percent commenting for senior high students is inflated due to the very high percent (90%) which commented on the "Walkout" commercial, compared with the other two ads. As was expected and supported by the data, senior high students were more negatively predisposed to anti-drug abuse information, and therefore counterargued more and "connected" less than the other two groups. Yet the "Walkout" commercial stimulated equal percentages of counterarguments for seniors and parents (52% of comments), and of the three ad stimuli, it produced the most counterargument for junior-high-school students.

COGNITIVE RESPONSES AND RANKING DRUG ABUSE AS SOCIAL PROBLEM BY SUBJECT POPULATION AND DISTRACTION LEVEL

As also expected, parents exhibited more connections than the student groups, when compared to seniors, and this is also generally true when compared to juniors. However, juniors exhibit the greatest percent of connections to the "Big Brother" commercial. Compared with other ad stimuli and with other respondent groups, there is no doubt because "Big Brother" appeals to juniors by portraying a junior-high-aged boy interacting with his older brother. Interestingly, while the ad was intended primarily to influence seniors, only 7% of their comments indicated connections with "Big Brother." Connections for this ad were higher among the other subject populations. Yet in any case, "Big Brother" stimulated the least counterargument among seniors, compared with the other two ad stimuli.

The ad stimuli also produced different effects for subject populations in stimulating concern for drug abuse as a social problem, and in stimulating interest in receiving more detailed information. As expected, from a list of 15 social problems, drug abuse was ranked as less of a problem by seniors than by juniors and parents. While "Walkout" was a powerful ad, judging by recall and comment levels, it was not very effective in stimulating drug abuse ranking for juniors and seniors; however, it was most effective among its intended audience--parents. "Walkout" also had most impact on parents' desires to receive a booklet containing information about drug abuse, relative to other ads, and relative to the other subject populations. Interestingly, while "Walkout" had powerful effects on cognitive responses among seniors, but little effect on ranking drug abuse as a social problem, the ad was most effective for this subject population in stimulating desire for a booklet. The latter results must be interpreted with caution, however, since the effects on booklet interest only approached significance (F=2.01, 4 and 357 df, p. <.10).

By examining effects of distraction (Table 2), it appears that percent commenting and cognitive responses generally decrease as distraction increases--note that effects are particularly strong in the "high" distraction condition. Adults are particularly sensitive to distraction, perhaps because their making cognitive responses require some effort, which is particularly difficult to justify when the information conflicts with existing attitudes. Consequently, for ad viewing conditions with high distraction, adults simply "tune out" much more than do junior- or senior-high students. However, the ranking of drug abuse as a social problem markedly increases with distraction among adults--yet the ranking decreases for junior-high students, and changes very little for seniors. It would seem that adults "compensated" for their diminishing cognitive responses by ranking drug abuse as a more serious social problem.

It was expected that increasing distraction would disrupt counterargument production among senior-high students, who were expected to be most negatively predisposed to the messages, to be most motivated to counter-argue, and to have most information with which to counterargue. However, the results indicate counterarguing decreases only slightly for this audience group. Perhaps distraction cannot effectively overcome relatively strong orientations regarding drug abuse among senior-high students. For Junior-high students, distraction has little effect--in fact, counterarguments slightly increase with distraction, contrary to prediction. It may be that the curiosity of these younger students with the topic of drug abuse is sufficient to overcome distraction.

SECOND STUDY

Research Procedures Used in Study Two

A second large-scale experiment was conducted in Bakersfield, California (n=753). The objectives of this study included testing reliability of the Palo Alto results, and examining effects of new variables [In the Bakersfield Study, a "competing messages" condition was implemented, in which subjects saw anti-drug abuse messages in the same program segments with over-the-counter drug ads, or other public service messages. The cover story, test procedures, audience populations, etc., were the same as in the Palo Alto experiment. However, based on the Palo Alto results, one level of distraction was used, midway in decibels between the "low" and "high" distraction levels in the earlier research.], including measures of behavioral consequences of exposure to the ad stimuli. A booklet about drug abuse was mailed to all subjects participating in the Bakersfield study within a few days after their participation in the experiment. Then, in follow-up telephone interviews conducted 10 to 14 days later, a sub-sample of adults (n=242) was asked if they recalled receipt of any booklet and is so was it helpful to them, did they read most or all of it, and did they have a drug discussion during the week?

Regarding reliability of the results, data in Table 3 compare the Palo Alto and Bakersfield studies in terms of nine dependent variables, for "Walkout" and "Big Brother." While absolute levels differ for the two studies, the patterns and directions of differences are consistent, except in two cases: percent of connections, and ranking of drug abuse as a social problem. For example, "Walkout" stimulated fewer connections (17%) in Bakersfield, compared with the respondent connections (29%) in the Palo Alto study. Nonetheless, the overall patterns indicate a reasonably high degree of reliability for the measures and procedures.

Effects of Competing Messages. The data show that competing messages affected gross recall, compared to the control condition of no competing messages (Table 4). When no competing massages were run, about three-quarters of the subjects recalled the three test ads. However, when public service or over-the-counter (OTC) drug ads were "competing," recall was heightened for the "Big Brother" and "Walkout" commercials, but not for the more neutral "People" commercial.

Examining the data more closely (Table 5), apparently the points recall for "Big Brother" and 'Walkout" was affected by the presence of competing ads, although points recall for 'Walkout" when OTC drug ads were present is slightly less than when no competing ads were present. However, points recall markedly increased when other public service ads were present.

Cognitive Responses: Counterarguing and Connections. Competing message conditions did not significantly affect counterarguing and connections. Moreover, competing message conditions did not produce differences in behavioral acts following exposure (e.g., requesting booklet, reading it, etc.). It appears, then, that "competing" messages simply increase recall, perhaps because the "competition" sensitizes people to the point made in the anti-drug abuse message.

Data in Table 6 show various behavioral consequences of exposure to the various ad stimuli. All of the sub-sample (242 adults) had been mailed in a booklet (whether interest in receiving one had been indicated in the post-test interview or not). Yet, however exposure to the booklet actually affected behavior depended on the particular ad the adults had seen in the experiment, 10 to 14 days earlier. That is, the test showed that compared to the control condition (subjects who had not been exposed to any of the ad stimuli in the experiment), those who had seen "Walkout" were far more likely to recall receiving the booklet (91%), and they were more likely to report having had a discussion about drugs in the ensuing week. (Due to timing of the phone interview, the discussion may have been a direct consequence of experimental exposure, i.e., occurring in the time between participation in the experiment and receipt of the booklet by mail). On the other hand, subjects who saw "Big Brother" were least likely of all subjects to recall receiving a booklet, but they were most likely to report having read most or all of it, and to have found it "very helpful." Fewer of these subjects reported having a drug discussion compared with those who saw the "Walkout" commercial.

The final stage of the project involved a field experiment in which "Walkout" and "Big Brother" commercials were broadcast over a split-cable television facility in a west coast city. [The facility is run by AdTel, Inc. As illustrated in Figure 1, there were 209 simultaneous showings of the two commercials on Cable A and Cable B over a 32-day period. This averages 6.5 showings per day, equivalent to an $18 million yearly advertising campaign. The cables reach highly comparable samples in terms of demographic characteristics.] In terms of the project objectives, the goal of this study was to test the response patterns obtained in the artificial laboratory environments to determine if these patterns would be validated in the natural conditions afforded by a field experiment. Space considerations in this paper preclude full analyses of the procedures and results for the field experiment. Data in Table 7 essentially confirm the earlier laboratory analyses of relative differences between "Walkout" and "Big Brother" (see Studies One and Two) in terms of the effects of these two commercials on adults. Midway during the field test campaign, following essentially equal exposures to the two ads, "Walkout" was better recognized, stimulated recall of seeing any anti-drug ads (even if "Big Brother" were not recalled specifically), incurred more negative affect and counterarguing, and produced fewer connections than "Big Brother." In every case, compared with "Big Brother," "Walkout" was associated with relatively greater incidence of behavioral effects, e.g., reading about drug abuse, talking about the problem. These results were highly consistent with the Palo Alto and Bakersfield studies. [In the post-communication campaign wave of interviews, most effects were found to converge for the two ads, to pre-campaign levels, or lower. Intriguing questions for research remain concerning these post-campaign effects, resembling, in a general way, the well known "sleeper effect". The topic is essentially important when stimuli are repetitive, as in the case of advertising.]

In this project report a summary has been presented of the rationale, procedures, and key findings of a major research project. We feel the study suggests some important areas of relevance for application in the public health area, as well as other managerial areas.

First, the reliability, validity, and sensitivity of the technique described in the two laboratory studies are strengthened by the field experiment results. Existing pretesting facilities could readily adapt the procedures documented in this research.

We feel the data argue for the procedures illustrated here. While further research is needed to more precisely assess the effects of distraction and competing messages, these manipulations are representative of real world viewing, and they affect responses to advertising in important ways.

We also feel the lab results regarding cognitive responses during communication--counterarguments and connections--suggest that communication planners should be aware of these message effects. For example, a message could obtain a high day-after recall score in a test market, but people may recall the message because they counterargued against it.

We have not meant to imply that counterarguing is always "bad". In fact, depending on campaign objectives, counterarguing could productively "set people up" for messages run in the next campaign wave which effectively refutes these counterarguments.

Finally, we would argue for the kind of "selection scheduling" pretesting illustrated here, in which multiple cognitive and behavioral responses are measured. The kinds of responses should be beyond simply asking for people's attitudinal reactions to test ads. Such measures are highly reactive and, in any case, people cannot reliably assess an ad's impact on their future behavior.

Depending on campaign objectives, somewhat different dependent variables could be employed which are different from those used here. Furthermore, we feel the procedures and measures could readily be adapted for use with materials other than finished commercials--perhaps roughs or even storyboards. The procedures could also be adapted for use with advertising in other media.