Bush Administration Very, Very Quietly Releases Abstinence-Only Study

RH Reality Check notes that the Bush administration chose Friday — the traditional day to release news that you’d rather the public not hear about — for releasing a new, thorough, Federal study on the effectiveness of abstinence-only education. And they released it without issuing a press release or advisory. (The report is available online as a pdf document).

These abstinence-only programs cost taxpayers $87.5 million dollars every year. So how much of a difference are they making? I think the pictures tell the story. The blue columns show how the kids in abstinence-only programs did. The white columns show a control group.1

No effect at all. None. Other than the fact that $87 million a year has disappeared from our collective pockets just as surely as if the money had been thrown into a furnace, the abstinence-only classes might as well have not existed at all.

No wonder the Bush administration is trying to bury these results.

“The rigor of the experimental design derives from the fact that, with random assignment, youth in both the program and control groups were similar in all respects except for their access to the abstinence education program services.” [↩]

No effect at all. None. Other than the fact that $87 million a year has disappeared from our collective pockets just as surely as if the money had been thrown into a furnace….

Uh … nope. If the money had surely been throw into the a furnace, I’d be confident it wouldn’t magically reappear in some right-winger’s pockets.

Bill Clinton sold presidential pardons and sleep-overs in the Lincoln bedroom. In contrast, Bush has sold anything that wasn’t nailed down. So every time I see a government boondoggle — earmarks, no-bid contracts, government works programs for graduates of Liberty University, outsourcing of military services to people who charge much more than our soldiers do, etc. — I have to wonder. To be fair, I can’t assume that people are crooks when they might simply be stupid. But I’m not assuming the opposite either.

No effect at all. None. Other than the fact that $87 million a year has disappeared from our collective pockets just as surely as if the money had been thrown into a furnace, the abstinence-only classes might as well have not existed at all.

Thats an interesting report. I wonder though, if we could look at the growth of the entire federal education budget in the last 6 years (or 10 – 15 years) and come to the same conclusion (money was thrown into the furnace, etc.)

I am also surprised. I would have expected to see worse results for the abstinence-only program, at least in the ‘Had sex, always used a condom’ category. I guess this shows that even those attending the program can get education from other sources.

FormerlyLarry, there are many ways you can study how well schools are doing.

But you can’t do a controlled experiment of this type, because we can’t run a test in which we randomly assign children into “go to school” and “don’t go to school” groups. Nor is it politically viable to assign schools randomly into “receives federal aid” and “receives no federal aid” groups. So a random controlled experiment of the sort which was run here — which is really the best way to test for things like this — probably wouldn’t be possible for the entire federal education budget.

Was the control group “no attempt at any kind of sex education at all?” Or was it the kind of “sex education including the mechanics of birth control and condom use for disease prevention” they used to teach in some schools 20-30 years ago? Or was the control group taught “sex ed as designed by Scarleteen” http://www.scarleteen.com/about.html (or similar groups)? they start from an assumption that everyone will masturbate, and almost everyone will want to start exploring sex before marriage, and they talk about how to for ensure one’s own physical and emotional safety in that exploration. It’s so much about empowered control of your own body and your own life it seems like the complete opposite of abstinence-only teachings.

I don’t recall my school’s sex ed sucking terribly much, but my most vividly instructional memories on the topic are all of the MTV “don’t assume just ’cause he/she’s hot that he/she doesn’t have an STD” MTV-style [redundant on purpose] PSA. I’m thinking of the eighties, though, when the band was playing on and all; do they still talk about safe sex on tv in a helpful and informative kind of way?

What also didn’t hurt was that the condom/banana lesson has become sort of a cultural stock meme, and iirc popped up in more than one sitcom or whatever when I was growing up. Come to think of it, I’m pretty sure I had a decent handle on the precautionary measures one needs to take with sex well before I really grasped how the whole sex thing technically worked. I didn’t wait for marriage, certainly, but I did wait until I felt like it, and understood and liked my reasons for feeling like it; that seems like a much better, far less arbitrary cutoff than Married (unless your goal is to get everybody married and locked in before full adulthood can kick in).

So I think by the time they’re getting the AOE, most kids are already of a pretty good mind about whether or not they want to have sex and when. A lot of ‘em also probably pay much closer attention to what pop culture tells them than teachers anyway, so in many ways I think the sort of ‘peer pressure’ effect of sticking useful info into America’s “I don’t feel like learning right now” face in between beer ads.

I note that teen pregnancy rates are not included in the study, only data on the unprotected sex & number of partners. Anecdotally, the rate quadruples when abstinence-only programs replace comprehensive sex ed. What I gather is that sex ed overall doesn’t much affect what teens do, but it does affect how they handle it when things go “wrong” — if the condom breaks, if a young man (referral to crisis center) or woman (referral + Plan B) is raped, if a a young woman finds herself unexpectedly pregnant, etc.

Was the control group “no attempt at any kind of sex education at all?” Or was it the kind of “sex education including the mechanics of birth control and condom use for disease prevention” they used to teach in some schools 20-30 years ago? Or was the control group taught “sex ed as designed by Scarleteen”…

It was a bolt on program. The control group is given what they normally have, and the program group what they normally have plus abstinence ed. It’s misleading to say these kids were given abstinence-only education. They were given standard sex-ed and had abstinence education on top of that.

Adrian, the control group consisted of kids at the same schools but in a general health class. There were four schools, two urban and two rural, and the two rural ones had a health class that lacked sex ed altogether (it didn’t cover STDs, contraceptive use, abstinence, sexual activity). The urban ones did cover sex ed to some degree (stages of reproduction, contraceptives, STDs, and abstinence as an option).

If you want to read or skim the study for yourself, that’s all on pdf p34. The breakdown of the four schools and their abstinence-only programs is on p15.

Someone on another blog noted that the study only shows *unprotected sex* statistics, not teen pregnancy stats. Other studies have shown how much worse AOE is for that–because while the numbers using condoms may not vary between the groups, the way they use them might; teens who understand fertility & risks are more likely to use condoms when it counts.

But you can’t do a controlled experiment of this type, because we can’t run a test in which we randomly assign children into “go to school” and “don’t go to school” groups. Nor is it politically viable to assign schools randomly into “receives federal aid” and “receives no federal aid” groups. So a random controlled experiment of the sort which was run here — which is really the best way to test for things like this — probably wouldn’t be possible for the entire federal education budget.

You can’t do a controlled experiment, but is already a running experiment which compares schools without federal interference with ones that do have federal interference. It’s called “private schools”.

Private schools have higher standards for graduation, stricter academic standards, and offer more advanced courses than public schools.

I’m always amazed at the amount of insistence of reenforcing abstinance-only teachings. I think if people were truly concerned with children and their education and well-being, they would take that money and invest it in better sex-ed programs. Teaching kids to protect themselves is a wise decision, though many people refuse to acknowledge it because of the whole irrational fear of increasing “promiscuity.” Regardless of this, why not ensure that people know how to protect themselves when they’re young? And since abstinance-only teachings don’t seem to make a difference in comparison to a control, that money should be used for sex-ed programs (and I am not referring to scare-tactics).

This is nice and all but if you think about it this program cost each tax-payer about $0.29. I would say it’s worth 29 cents to find out that abstinence isn’t the answer. At least their is proof that abstinence is not an answer.

Oh my Gawd! Abstinence only education doesn’t work as advertised?! What are we, as a society, to do to keep teens fearful of sex? Time to ramp up the sex predator moral panic to a fever pitch. Quick, someone call Chris Hansen and Dateline NBC!!

I’m 5 years removed from being a teenager, and so I look with skepticism on any results whatsoever taken by surveying teens about sex. Between what we wouldn’t admit to our parents that we did, and what we wouldn’t admit to our peers that we hadn’t done, I don’t think there’s anything left that we’d be honest about.

It seems to me that lying is very much a recognized problem with surveys of teenagers. On the other hand, it doesn’t seem like it would matter much in this case, unless we had reason to think that the two groups would lie at different rates (not likely).

Well, duh. Abstinence works, but they obviously forgot to concentrate heavily enough on slut-shaming, homophobia, and Biblical mandates regarding marriage. those are necessary subtexts for successful abstinence teaching, and the damn liberals interfered with it. Obviously the programs WOULD have worked, they just got ruined by liberal influences. What’s the problem with that? We should just run them again, but do it right this time.

(remove tongue from cheek)

heh. that’s pretty fucking hilarious. I mean, ZERO change? NONE? After all that fuss about how it was going to save our youth?

I’m not sure why you think these results are so worthless, or why the money was wasted. For a lousy $87.5 million (only $0.25 of which you had to pay for), you now have a peer-reviewed argument to deploy in political fights.

I’d gladly pay $0.25 for a study that proved (say) that single payer health care led to worse overall outcomes. You should be HAPPY.

Oy. Bush is such an arsehole.
I remember reading something last year in the Wall Street Journal about a pregnancy-prevention program in some rural part of North Carolina (I think). Most of the people in this area were very poor, and there were a lot of children who had been raised in single-parent homes, with mothers who had given birth to them as uneducated teenagers. However, it wasn’t abstinence-only sex education. If I remember correctly, the educators presented abstinence as THE BEST way to prevent teenage pregnancy, but ALSO made a community-wide effort to have free condoms availible to teenagers and to make sure that the teenagers knew how to put them on. THAT is the sort of program I could read about and say: “You people know what you’re doing.” Bush’s abstinence-only shit? Not at all, especially since it said in the article that teen pregnancy rates in North Carolina were dropping since the inception of these programs, but according to this chart…nothing changed anywhere.

I have a friend who will be a virgin when she gets married this June and my brother is also in a long-term committed relationship with a a woman who plans to remain a virgin until marriage. What has stopped these women from having sex thus far was not so much the facts about intercourse that they learned in school but the values and morals concerning sexual relations that they learned at home.

Both women are from religious households.

I think a part of the reason that abstinence education is shown to have had a negligable effect in this study might because the parents aren’t talking to their children about the why of avoiding sex and how to avoid it until your ready. In other words, the parents think that because their kids are learning the facts at school and being told “just don’t do it,” that they ( the parents) don’t have to do much talking about relationships at home. Parents are a big part of encouraging children to remain abstinent until marriage and if you remove one part of the equation ( either the facts or the morals/values) then you will get failure.

These abstinence-only programs cost taxpayers $87.5 million dollars every year.

That’s not the cost of the study. That’s the cost of the seemingly worthless program every year, yes? I don’t think it’s the study that’s being called worthless. Admittedly, you couldn’t have the study without the program but…it’s a pretty high price to pay for something with no effect.

You guys are seriously misguided if you think the only implication of this study is that abstinence education doesn’t work. Note that the results are pretty much equal across the board. As number 18 points out, that includes rural high schools with basically no sex ed, and urban high schools with traditional sex ed.

So what this really proves is that nothing — including traditional sex ed, or a complete lack of sex ed altogether — makes a difference one way or the other.

Proponents of abstinence are no doubt dismayed. But if they were thinking clearly, proponents of traditional sex ed would be equally dismayed.

On a different note: The money we pump into AOE isn’t limited to here in the US. By law, 33% of all the money in PEPFAR (President’s Emergency Plan for AIDS Relief) is required to go to AOE. So we not only push an ineffective program here in the US, but use our power to push it all over the world as well…. Last year, the PATHWAY Act (HR 1713) was proposed to repeal the 33% earmark. It garnered a good deal of support but didn’t make it out of committee.

P.S. In response to John Doe: I find these findings puzzling for the reasons you’ve cited. In fact, some of these findings contradict a whole decade of academic research. Most studies have found that students limited to AOE are less likely to use a condom or contraceptives when they choose to have sex. A good overview study (though a bit dated) is by Grunseit, Kippax, Aggleton, Baldo, Slutkin, called “Sexuality Education and Young People’s Behavior: An Overview of the Studies.” It’s in the the Journal of Adolescent Research, Oct. 1997. However, the lack of improvement under AOE does seem problematic for the current administration, after they’ve done so much to tout it.

No effect at all. None. Other than the fact that $87 million a year has disappeared from our collective pockets just as surely as if the money had been thrown into a furnace, the abstinence-only classes might as well have not existed at all.

But if you really believe the study’s results, the entire $206 million (cite) in federal funds is “thrown into a furnace,” as is however many millions of dollars that states themselves spend.

It’s really not fair to take a study that shows abstinence education = traditional sex ed = no sex ed at all, and then claim that this only proves that abstinence money is wasted.

It’s really not fair to take a study that shows abstinence education = traditional sex ed = no sex ed at all, and then claim that this only proves that abstinence money is wasted.

The study doesn’t prove this. It proved ‘whatever they normally get’ + ‘abstinence education’ = ‘whatever they normally get’. The program group got extra abstinence programs on top of whatever education they normally got. It was a specific assessment of whether the $87.5 paying for these extra programs was doing any extra good, and it wasn’t.

Lots of people are throwing around the phrase ‘abstinence-only’ in a vague manner. Because of that I originally thought these kids got abstinance-only eduction in the sense that that’s all the were taught: how to be abstinent and nothing else. In fact they got normal sex-ed and abstinence-ed. So it’s a bit misleading to say these kids were in abstinence-only programs, when they were also having class teaching them about how to use condoms and stuff.

Proponents of abstinence are no doubt dismayed. But if they were thinking clearly, proponents of traditional sex ed would be equally dismayed.

I don’t necessarily follow your logic, considering proponents of traditional sex ed aren’t promoting abstinence, but instead are promoting safe sex. Granted I need to read the study, but from the charts alone, more kids are using condoms than not which seems like at least a small victory to me.

Thanks for the article, a good read. The problem with abstinence-only extends out beyond the US unfortunately. In 2005 Uganda adopted such a program, under US funding. Over there though, these programs result in death from increased HIV infection rates. These programs are criminal and must end. I wrote about this story in my blog.

It’s really not fair to take a study that shows abstinence education = traditional sex ed = no sex ed at all, and then claim that this only proves that abstinence money is wasted.

That’s not what it says at all. Or, at least, that’s not what the above graphs say. It doesn’t separate out whether the students came from the sex-ed schools or the no-sex-ed schools. What it actually says is that:

The most intuitive way to interpret that, and the way most people in this thread seem to be taking it, is:

traditional sex ed AND abstinence only = traditional sex ed
no sex ed AND abstinence only = no sex ed

which is not necessarily accurate at all. For example, abstinence only education could be having an averse effect on one group and a positive effect on the other, but the two effects are equal (given equal sample size from each group) and cancel each other out.

Much as I would like to say we should use this to try to end abstinence-only education (because I really do think it’s a stupid thing to be teaching kids – sex happens, and always has, which is why many of our ancestors were married at 13), and while I do find the current graphs are nice and illustrative that there is some point to be made, I don’t want to twist around statistics to prove points to ignorant people. Really, what we need to see is this same histogram as it would appear for each of the schools tested before we can draw a real conclusion (skimming through the actual study, that looks to be impossible, because for some of the schools, especially the ones with no sex-ed, it looks like the abstinence-only stuff was mandatory). If that looks similar, then we very much have a case.

Actually, if you look far enough down in the data, there was at least one effect: students who took the “abstinence only education” were less knowledgable about condoms and what they will and won’t protect you from than controls. Also the AOE students were slightly less likely to have smoked in the last month. I also notice that 2/4 programs were voluntary, which probably skewed the results somewhat. I would expect that participants in the voluntary programs were likely to be more favorably inclinded towards abstinance in the first place, suggesting that AOE might actually discourage abstinence. I also wonder how many of the “abstinent” responders might have been engaging in oral or anal intercourse. Or lying.

I’d gladly pay $0.25 for a study that proved (say) that single payer health care led to worse overall outcomes. You should be HAPPY.

To give proper credit, they did do the study. That’s certainly worth something.The problem is that, as far as I can tell, the Bush admin has no intention of making any policy changes based on the results. Would you gladly continue to pay $0.25 (or $87 million) for continued single payer health care after a study demonstrated it was no better than or worse than the current system? (An unlikely result, but postulating for the moment that that would be the outcome.)

james, did you actually read the report, and its definition of “abstinence-ed”?

Yes. Did you skip the section on study design? Although these kids were receiving abstinence-ed classes (the treatment) they were otherwise identical to kids in the control group, and were receiving the same standard sex-ed as them.

I worked behind the scenes to help write the Scott & White Worth the Wait® abstinence curriculum (c. 2003, Grades 6, 7, 8, and High School — http://www.worththewait.org). Part of my job was to research the most recent medically and psychologically based studies, summarize them accurately, document them correctly, and implement the factual information into the curriculum. Each word in those four notebooks passed under the watchful and critical eyes of Patricia J. Sulak, M. D., the director of Scott & White Worth the Wait® in Temple, Texas.

Dr. Sulak is a well-known ob/gyn specialist and contraceptive researcher at Scott & White Hospital and Clinic. She is confronted on a daily basis by patients who are experiencing the painful consequences of early sexual activity, and she is totally dedicated to encouraging teens to remain abstinent until marriage.

Because of Dr. Sulak’s desire to remain medically accurate, she and her staff insisted on measuring the results of Scott & White Worth the Wait®; and the results have been published in American Journal of Obstetrics and Gynecology, a peer-reviewed medical journal. The results demonstrated “a knowledge gain and a positive shift in attitude towards delaying the onset of sexual activity.” This research utilized not only self-disclosed student responses (as done by Mathematica in their report) but also formalized medical data from the Texas Health and Human Services Commission.

A major problem with the Mathematica Policy Research report is that credible abstinence programs such as Scott & White Worth the Wait® were not chosen to be a part of the study.

During the 2 ½ years that I worked for Dr. Sulak, I became quite familiar with the leading Title V, Section 510 abstinence-education programs; but I have never even heard of the four programs that Mathematica chose for its study. Mathematica did not choose well-established and well-researched abstinence programs (e.g., Scott & White Worth the Wait®, Best Friends, Not Me/Not Now, Operation Keepsake, Abstinence by Choice, Teen Aid, Sex Respect, FACTS, Project Taking Charge, etc.). Evidently these programs did not fit the Mathematica research model.

To help the public understand what the Mathematica Policy Research report is and what it is not, I have included important statements from the actual report along with my own personal comments:

These are the only four Title V, Section 510 Abstinence-Education Program grantees which were studied: (1) My Choice,My Future! in Powhatan, Virginia; (2) ReCapturing the Vision in Miami, Florida; (3) FamiliesUnited to Prevent Teen Pregnancy (FUPTP) in Milwaukee, Wisconsin; and (4) Teens in Control in Clarksdale, Mississippi.

These four programs were chosen because they represent diverse settings:

The evaluation team visited and observed 28 abstinence education programs across the nation. Eleven of these, representing a range of program models and serving different target populations, were invited and agreed to participate in the evaluation. This report focuses on 4 of these 11 programs. These four programs are called impact sites because they had program features and staff capable of supporting a rigorous, experimental-design impact evaluation.

What this means is that these four programs were chosen because they fit the specifications of the Mathematica Policy Research design. To say it another way, these four programs were chosen because they were situated in schools/communities which had the staff and the diversity which Mathematica needed to carry out the study.

Other programs with established and successful research to prove their effectiveness were not chosen because they did not happen to fit the research design.

For instance, it takes special circumstances and arrangements with schools and communities for a program to be able to set up control groups.

Because of state laws, it also takes certain circumstances for schools and communities to participate in a research study which asks very intimate questions of students. Many of the most respected abstinence programs in the United States are located in sites where such questions are not permitted by law and/or by parental groups.

I think it’s reasonable to presume that there’s more than one way to design an abstinence-only sex-ed program, and there’s more than one way to implement any given program. Apparently this study was based on a very limited sub-set of such programs, so presuming that this study’s results is an accurate measurement of abstinence-only sex-ed programs as a whole is not sustainable.

MSM reporting on scientific studies sucks, folks. They rip and read, but rarely seek or report critical analysis of information outside their area of expertise before they go to press.

Ron, have you read the Sulak article in AJOG? The program they used is a sex ed program, not an abstinence only program. They don’t say exactly what the curriculum of their program included, but based on other papers they have written (including a program to teach adults how to talk to adolescents about sex), it appears that their program emphasized abstinence as the best method of preventing sexual related problems, but discussed other methods–including what condoms and other forms of birth control will and will not do–as well. In short, if their program worked, it is a sign that sex ed is worthwhile, not that abstinence only ed is. As you said, MSM reporting on science isn’t the best and can be misleading. But so can editorials, even from experts who should know better. Read the original research.

Nope – haven’t seen the original article. I was past the time I had at the time. However, even if the one program she quotes is not an abstinence-only progam, the part of the criticism that says, essentially, “they chose only 4 of the 11 commonly used programs and excluded ones that has research backing up their effectivness” still needs to be addressed.

This report focuses on 4 of these 11 programs. These four programs are called impact sites because they had program features and staff capable of supporting a rigorous, experimental-design impact evaluation.

What this means is that these four programs were chosen because they fit the specifications of the Mathematica Policy Research design. To say it another way, these four programs were chosen because they were situated in schools/communities which had the staff and the diversity which Mathematica needed to carry out the study.

Other programs with established and successful research to prove their effectiveness were not chosen because they did not happen to fit the research design.

Yes, but Mathematica’s research design is the absolute best research design; random assignment into program groups and control groups, with a demographically diverse group of students. It’s not unreasonable for Mathematica to design their study to use the most rigorous methods, nor does it show that they were biased, except insofar as “best research design” is a bias.

If the “other programs” are not willing or able to be evaluated using random-assignment of a diverse group of students, then I question whether those “other programs” have really proven themselves to be worthwhile and effective.

Not that the mess the Bush administration created in 8 years is going to get cleaned up quickly, but at least it will *hopefully* stop getting worse once he’s out of office.

I’m beginning to second guess my decision NOT to move to Canada after the last election.

Why anyone with the least bit of common sense would think abstinence ONLY education is an effective tool for fighting STDs and unwanted pregnancy is beyond me. I am not coming at this from an academic perspective, but rather as one from my own experiences as a teenager and with Sex ed. Annecdotal evidence/personal experience opinion all the way – take it for what it’s worth.

I had pretty comprehensive Sex Ed. Age appropriate sex education began in 5th grade and continued through high school. I didn’t run out to have sex because of it, nor did most of my peers. In fact, I didn’t have intercourse until I had graduated college – and I wasn’t the only one of my peers to do that and none of us are/were particularly pious and religious. But the ones who WERE having sex (who would have done so anyway more than likely given historical estimates of rates of teen sexual behavior) certainly were having safer sex than they otherwise might have.

At one point, the student council of my HS lobbied to put condom machines in the school bathrooms at the high school – they made a compelling argument (which was squashed of course), but the fact that they were even able to voice the argument to begin with was pretty good progress I think – and that was over 10 years ago.

For the record, I’m not a person who thinks teenagers should be having sex. It may in fact be my most conservative viewpoint. I just don’t think before the age of 18-20 or so most people are really able to competently handle the mental, emotional and physical consequences of sex. And I plan on teaching my daughter about the importance of waiting until she’s really ready, but I’m also not going to refrain from teaching her about the importance of proper condom use and other birth control. I suspect I’ll even discuss/demonstrate the proper application of a condom – can’t say if I really will or not, she’s only two, so who knows, but I’d like to think I will. That being said, I think it is the job of parents to instill the confidence and sense of self worth required to be able to abstain from sex in adolescence.

The school’s job is to educate – and basic reproductive knowledge and understanding how to prevent disease and unwanted reproduction should be part of every basic Health class. Isn’t that common sense?

essentially, “they chose only 4 of the 11 commonly used programs and excluded ones that has research backing up their effectivness” still needs to be addressed.

The programs to be evaluated were chosen in advance and therefore could not be specifically chosen for their efficacy or lack thereof: no one knew apriori how effective they were likely to be. If you have any evidence that any other AOE program is more effective, please cite it because all the literature that I’ve previously read suggests that AOE is not just null value but acutally dangerous in terms of increasing the number of teens engaging in risky sex, without actually reducing the number engaging in sex of any sort. So if anything I am suspicious of this study because the AOE appears to do better than the literature suggests that it should.

Ron: A couple more points about the editorial you linked to:
1. The author complains that AOE with “proven efficacy” were excluded from this study, but she does not cite any literature about any AOE program demonstrating it to be effective. She only mentions one program which is NOT an AOE program as an example of effective programs (and given that the only papers on it suggesting that it is effective are written by the author of the program itself, I have some doubts about their general quality).
2. She goes on to cite data in the Mathmatica study that fits her agenda as factual. Either the study is fatally flawed or it is not. If it is, then the “good news” from it (ie that students who underwent the program are more likely to overestimate condom failure rates) is no more reliable than the “bad news” (ie that there is no increase in abstinence in the treatment group). If it is not, then both pieces of data are equally valid.

essentially, “they chose only 4 of the 11 commonly used programs and excluded ones that has research backing up their effectivness” still needs to be addressed.

Pretty much what everyone so far has said seems reasonable to me, but I’d also like to add that claiming, in effect, “Hey, so it’s just a coincidence that every program they chose to examine (over 1/3 of our total programs) was ineffective. The others are totally useful,” seems kind of . . . not . . . good.

I mean, “Some of our programs aren’t useless!” Isn’t exactly the greatest rallying cry in history.

If the “other programs” are not willing or able to be evaluated using random-assignment of a diverse group of students, then I question whether those “other programs” have really proven themselves to be worthwhile and effective.

Actually, what’s at issue is not that the programs refuse to be evaluated, but that the schools using the programs and the group of kids in them had to fit certain not-well-described study criteria.

I’d accept this study as valid for the programs and locations it covered, but extrapolating it to 7 other programs that weren’t looked at seems a bit of a stretch.

And for that matter, it’s a lot harder to get rid of a teacher, staff member or principal.

I’ve said it before, and I’ll say it again; the biggest problem with the public school systems is the home environment. Teachers and schools can help but ultimately cannot overcome the issues of children that come to school unprepared to learn.

My opinion is that abstinence education is a complete waste of money; that it has little effect, and who knows in which direction. It is also my opinion that this study by Mathematica Policy Research is fatally flawed. It is an example of amazing incompetence.

Didn’t it strike you as odd that all the results were the same?

The study involved a total of 2,057 students in both the control and program groups, about equally divided, which means that a random fluctuation in some result by ten students would shift the reported figures for a particular characteristic by one percentage point. Even if the populations were completely identical, one would expect variations of several percent in the results obtained based solely on chance.

Even if two samples were accidentally drawn from the same group one would expect variations of several percent in the results obtained based solely on chance.

In fact the study reported mostly identical numbers for both groups for nearly every characteristic—no variation at all, random or otherwise. This result could only be obtained by manipulation whether accidental or intentional.

Examples, the percentage or scale mean for the program group comes first, then the number for the control group: (some of this data was in the first post in this thread)

Assuming that 49% is the correct number for the populations that these groups represented, what is the probability that a random sample of a little over one thousand members would return a result of 49%?

This is pretty close to a coin toss, so just simplify the problem by assuming that exactly 50% of both populations were abstinent. If you threw a thousand coins, what are the chances that you would come up with 495 to 505 heads, inclusive?

And then what’s the probability that you would come up with 495 to 505 heads if you did it again?

More formally, as Newton explained over 300 years ago, you can determine the probability of coming up with 495 to 505 heads in 1000 tosses by expanding the binomial (x y)^1000, summing the coefficients of terms 496 to 506, and dividing by the total of the coefficients.

You would come up with 495 to 505 heads about 28% of the time. The probability of obtaining this result twice in a row when the sample size is 1,000 is .28*.28 = about 8% or about one chance in twelve.

But, in the first sixteen results reported in the Mathematica study, eleven were identical, four differed by one point, and only one by two points. The probability that these results would occur in random samples of this size is on the order of .08^16 or about 3 chances in 100 million billion (10^17) trials.

This is essentially an impossibility. The study results could not have occurred in the way described in the report.

I suspect that they screwed up and entered the same data twice. What little variation exists is because of other errors in data input, processing or rounding errors.

I find the whole thing kind of disturbing. Like the Emperor has no clothes.

I think that even if we like the results of this study, it’s better to know the truth–they screwed up royally.

In fact the study reported mostly identical numbers for both groups for nearly every characteristic—no variation at all, random or otherwise. This result could only be obtained by manipulation whether accidental or intentional.

You are absolutely correct. This is intentional, and they describe within the study report what the manipulations are: weighted regression and estimation.

First, the results are biased using weighting methods which cause the figures to appear to be less variable than they are. This is done to highlight statistically important variations, causing the variations you describe to disappear.

Second, thoughout virtually the entire study, and in particular in the Executive Summary where all the figures you refer to are found, you are not looking at the statistical data at all. You are looking at “Estimated Impacts”. The studies first collected the data, weighted the results to smooth variation due to random factors, then used these results to create *estimates* of the effectiveness of the results. This is what you are looking at.

This results in figures in which all unimportant statistical variances, such as the ones you rightly point out should (and do) exist in the data itself, are deliberately eliminated.

For a table that shows the variances in the actual data, rather than the estimated impacts, see pg. 112 of the original study, Table A.14: Impacts on Perceived Effectiveness of Birth Control Pills for Preventing Pregnancy, Overall and by Site. Note that these are still weighted figures. However, despite the weighted figures, this shows variances of up to 16% between the two groups.

well i personally believe that teaching abstinence only sex education is just a sense of ignorance. adults know that a lot of kids are having sex nowadays and to not inform them of the necessary things to protect themselves is even worse than just saying “go have sex.” tennagers probably shouldn’t be having sex but u cant expect them to just go along with what some silly program tells them to do. schools should talk about how students shouldnt have sex but if they do, here are the other options….its that simple. abstinence only education does not work!!!

Well, Kelsey, you can believe as you like, but the question at hand is whether or not that’s something that has been proven by this study. And while it was touted as proof that AOE does not work, closer examination indicates that there are a lot of questions regarding it’s validity that don’t have good (or at the very least, clear) answers.

well it may not say that its proof that it doesn’t work, however it is shocking that there is no proofs to back up the claims bush says about it being all worth it. Sometimes the absence of evidence is enough to prove something and I think that the people who made this site also made their point loud and clear!