Do better readers do better on tests of reading?

Yesterday, I took an initial look at the Manhattan Institute’s study, “Building on the Basics.” Today, I want to look at Florida’s state science exam, the focus of the study. A common criticism of standardized tests is that they all, to some degree, test reading ability. What does the Science FCAT look like? What skills would you need to perform well on it? I’ve only seen the NYS Science exams, so I decided to download a Florida sample test and take a look. The first thing that surprised me about this test was the reading level, which seemed high. Many of New York City’s fifth graders would (for better or for worse) stumble over sentences like, “Florida has many limestone caves containing formations called stalactites.” I tracked down a site of readability analyzers and entered text from test items.

Question 1: Melissa’s school rings a bell to alert students that it is time to start class. When the bell rings, it vibrates. The use of vibrations to send messages is an example
of which type of energy?

This one ranged from 4.72 to 10.07 in estimated US grade level required to understand it, which certainly calls into question the reliability of the readability analyzers, but also the ability of average 5th graders to understand this question.

This one ranged from 8.26 to 12.08 in grade level readability analysis:

Question 15: The Florida panther is an endangered species living primarily in and around the Everglades. Cows, raccoons, black bears, and bobcats also live in Florida. The teeth of these animals enable them to eat different things. A drawing of a Florida panther and a chart comparing the diets of these Florida animals are shown below.

Whatever the validity of the readability analyzers, these questions seem to be written at a sophisticated reading level considering that the test aims to assess content knowledge and reasoning rather than reading skill. Obviously, students who read better will perform better on this test, but does that correspond to better scientific reasoning or grasp of scientific knowledge? A fifth grader might stumble over words like “primarily” and “enable,” but have no problem with the concept that animals that eat similar foods tend to have similar teeth. What does the test really assess in this question?

Finally, while reading, writing, and mathematics are all important to scientific understanding and communication, high quality science instruction goes beyond what can be captured in a paper-and-pencil test. The Science FCAT includes some “read-inquire-explain” questions that ask students to demonstrate scientific thinking; for example, question 14 asks:

At the park, Matthew has collected 20 pill bugs from under rocks, logs, boards, and bricks. Matthew wants to conduct an investigation to find out if pill bugs prefer moist areas or dry areas. Design a procedure Matthew could follow to test whether pill bugs prefer moist areas or dry areas.

Being able to design a procedure is important, but this question makes me wonder how many of Florida’s students who attend F-sanction schools get to go beyond hypothetical experiments and do real ones. Unlike New York’s Intermediate Level Science Exam, which includes a more authentic “performance” section in which students conduct hands-on investigations and draw conclusions from data they gathered themselves, Florida’s test is all reading and writing. While students may perform somewhat better on this test, the danger remains that other aspects of science are still being “crowded out” of the curriculum as they are not assessed by the state at all.