April 25, 2007

Accelerated Reader gets the WWC treatment

This report follows an alarming pattern in which the WWC allows the evaluation of the research to include a testing instrument developed by the authors of the education program as a valid measure of success. Out of the 35 studies reviewed, only one (Ross, Nunnery, & Goldfeder,2004) met WWC standards. And guess what:

The STAR Early Literacy test and STAR reading test are the only outcomes reported in the study. The STAR tests are developed and distributed by Renaissance Learning, which also distributes Accelerated Reader/Reading Renaissance.

The test is a computer-adaptive, norm-referenced test that measures student reading comprehension. It is designed for students who have at least a 100-word reading vocabulary and can be used with all students in grades 1–12. Students read passages of text and fill in key missing words from a set of options (modified cloze procedure).

What's surprising is that even using their own testing instrument, Accelerated Reader didn't perform all that well.

Comprehension. Ross, Nunnery, & Goldfeder (2004) reported a positive and statistically significant effect of Accelerated Reader/Reading Renaissance on third grade student performance on the reading comprehension measure (Star Reading test). In WWC computations, this positive effect was not statistically significant, but considered substantively important according to WWC criteria (an effect size greater than 0.25).

General reading achievement. Ross, Nunnery, & Goldfeder (2004) showed that Accelerated Reader/Reading Renaissance had positive and statistically significant effects on the general reading measure (Star Early Literacy test) for kindergarten, first, and second grade students. According to WWC analysis, the average effect size across grade levels was statistically significant.

With respect to general reading achievement, to get statistically significant results, the WWC had to average the performance of grades K-3. None of the grades individually had sufficient students to achieve a statistically significant result.

Yet on the basis of this one small study using a questionable testing instrument, the WWC concluded that Accelerated Reader has "potentially positive effects" for reading comprehension and general reading, earning the coveted green box and +?.

This seems like extremely flimsy evidence to me and it seems to send a message to publishers on how to cook the books to get the thumbs up from the WWC.

13 comments:

Are the studies ever repeated? While what you bring up is certainly grounds for criticism, the real test would be whether other experimenters can reproduce similar results with other measures or samples. A single study is nearly always flimsy evidence in social sciences. Even though the measure was designed by the developer of the program in this case, one would need a better reason than that to call it invalid.

First of all Observer, you miss the point entirely. The question is not of potentially bised researchers, it is that the researchers, whether they were potentially biased or not, were using a potentially biased testing instrument. I didn't even reach the potentially biased researcher question.

But let me answer your question anyway. I do not know about the research base for SfA or OC. but, I do know the research base for DI. About have the studies were conducted by affiliated researchers. The results of those studies were about the same as the studies conducted by independent researchers. Furthermore, as far as I know all the data from the studeies are available for inspection, so it is a pure cheap shot to criticse the researchers in the absence of some evidence of actual wrongdoing.

"This report follows an alarming pattern in which the WWC allows the evaluation of the research to include a testing instrument developed by the authors of the education program as a valid measure of success."

If your point was about testing bias as opposed to researcher bias, the first sentence of your post doesn't reflect that.

Do you know whether Renaissance has allowed it's analysis to be open for inspection? I would agree with you that that's fishy if it has not.

BTW, there are no talking points. I am very, very removed from the stakes here. But you know, not everyone who disagrees with you or questions you is a puppet being blindly lead down the path.

That's just zealotry. Part of "scientifically-based" is not just a set of principles, but a mindset, a way of viewing the world that is objective and grounded in evidence.

"This report follows an alarming pattern in which the WWC allows the evaluation of the research to include a testing instrument developed by the authors of the education program as a valid measure of success."

"developed by the authors of the education program" describes the testing instruments.

The sentence quite clearly refers to the test, and not the researchers.

I formally worked at a school where AR was touted as the save all, be all. If we didn't do anything else, our students HAD to do AR religiously. We also used STAR Reading and STAR Math as our primary assessment tools (aside from the state assessment). What we found out is that both STAR Reading and STAR Math were not reliable measures of reading or math proficiency.

Secondly, AR is a reading comprehension assessment. Students read trade books and take short comprehension tests. AR has NOTHING to do with the instruction of reading.

Whatever study somebody comes up with is irrelevant. If a child's reading did it improve, it was because of the child's repeated practice of reading. TAKING A WHOLE BUNCH OF AR TESTS MEANS NOTHING.

What happens is that some schools get caught on the AR carrot and stick reading bandwagon. Students "read" books for prizes and points. The entire reading experience goes from quality to quantity.

I'm curious as to whether AR would be good for kids to help them expand their repertoire of book reading, and to get them on a regular schedule of reading as an activity (rather than tv, games, etc). Not that it is available at home but that it might be helpful to replace some home "wastes of time" with something a little more cerebral.

For kids who don't like to read for pleasure and are forced to read AR books as part of a grade, AR is a disaster. For kids in which English is their second language and struggle with reading, AR is also a disaster.

Our school had AR reading fully implemented up until this past school year. I "taught" AR for five years. Our last period of the day for my sixth graders was devoted to AR reading. This meant that students were tested with the STAR test to determine their ZPD (Zone of Proximal Development) scores. Once we knew what their ZPD scores were, students checked out books from our library which is completely set up as an AR library.

During the last period of the day, students would read their AR books and when finished, they would go to the library to take their AR tests. At the start of every quarter, students were given a goal (points to be read) during the quarter. Points are determined based on their ZPD level. For instance, if a student was reading at a 2.3 to 3.3 level, their points for the quarter might be 20 points.

If the student reached 80% of the point total, then their grade would be a B-. As a person who LOVES to read, this last period would have been wonderful for me as a student. However, for students who don't like to read, those who struggle, or those who were second language learners, it was a system which set them up for failure.

I am so glad we are not using AR this year, especially for our struggling learners.

Our schools use AR, its an entirely extra-curricular program. Basically it rewards students who would read anyway. My biggest problem is that it has limited the books that my oldest daughter will read. She has forgone reading some books because they aren't part of the program and wouldn't earn her points.

That's another issue I have with AR (as Rory mentioned). At my school, kids were discouraged from reading books that were not in their AR range because ALL books that they checked out, they needed to take a test on. Now, I've always been a very voracious reader and when I was a child, I read what was interesting to me. I cannot imagine how horrified I would have been as a child had I picked out a book that looked interesting, only to be told "no" because it was out of my reading level.

My son was a reader, but had a very limited scope (only non-fiction war stories). AR helped him expand his horizons greatly.

He loved the competitive feel to it and enjoyed the goofy rewards. He read more books cover-to-cover than I think I read in all of grade school. He was exposed to different authors that interested him so much that he continued on with their series later. He discovered the joys of fiction (I can hardly get him to pick up a non-fiction book anymore.)

Reading books is a habit and requires a stamina of sorts. He developed both by way of AR. He was a pretty voracious reader all along, but the habit he formed only solidified.

I was surprised because I think I would have been irritated by it myself. I came along in the Boomer days (SRA color-coded cards, anyone?) which I despised. However, I spoke to a friend who remembered them and absolutely loved them.

I see the caveats with such a program, so maybe as Rory stated, using it as an extra-curricular program might be better idea.But it does have its positives for certain kids. I'd hate to see the baby thrown out with the bathwater.

Another thing, our teachers did not prevent kids from going out of their age range. They just gave a little warning about it. They also told them to not to take the test results too seriously. It became more of a game with the kids and less of a boring chore. That probably helped.

Susans, I think you're right if it's not used as part of a grade. In my son's 8th grade English class, AR was 20% of his grade. As I've mentioned before, he has never developed a love for reading and so this was pure torture for him.

If you use a hammer to snake your clogged toilet, blaming the hammer for the resulting disaster seems a little silly, right?

Such is the case with AR. It is not instruction. It is not a class. It is not an approach. It is not a set of strategies.

AR is a system of tracking and accountablity for the type of independent reading most would agree is necessary to develop and maintain strong reading skills. It allows students to locate a skill-appropriate reading range (ZPD), take tests upon the completion of books, and easily track the level of books read, the type of books read, levels of success, and so forth.

It does not supplant, nor should it supplement instruction.

I find AR invaluable because my (90% ELL/ 20% SpEd) students need to read on their own, utilizing books wherein they can access the vocabulary and content, but I consider my insructional time to be far too valuable to devote to this task. AR provides the accountability tool I need to ensure that the reading is happening, and is happening well. I do make the acquisiton of points a part of their grade, and I also monitor book reading to ensure that once a standard of mastery is met, students' ZPD is raised, and more challenging books are read. I also allow some flexibility base on high interest, etc. When kids struggle to pass quizzes, we talk about strategies, and in the face of continued failure, lower ZPDs. When kids do not read, I enroll them in an after school reading "class" in which I monitor their reading and quiz taking.

AR is not perfect. It's leveling criteria fall solely on a measure of vocabulary, where other factors contribute to the difficulty of text (shifting narrators, backward plot design, multiple subplots or parallel episodes), but all the AR "horror" stories make it clear to me that teachers/ schools/ and districts are the ax-weilding psychos, not the tool itself.

As for reading being "torture," I felt the same way about geometry. Oddly enough, the school still insisted I learn essential content area knowledge and skills, regardless of my personal preference. Weird.

About D-Ed Reckoning

The primary problem with K-12 education today is the problem of dead reckoning--an estimate based on little or no information. We don't know what a good K-12 education system is because we've never seen one operating. A good education system is one that is capable of educating almost every child.