Lens on Science: Development and Validation of a Computer-Administered, Adaptive, IRT-Based Science Assessment for Preschool Children

Year: 2009

Name of Institution:University of Miami

Goal: Measurement

Principal Investigator:Greenfield, Daryl

Award Amount: $1,570,265

Award Period: 4 years

Award Number: R305A090502

Description:

Co-Principal Investigator: Randall Penfield

Purpose: Recent emphasis has been placed on science as a school readiness domain. However, an obstacle to conducting research and evaluation on early childhood science programs is the lack of appropriate, reliable, and valid direct assessments of children's science knowledge and process skills. This measurement project will fill the need for a reliable and valid direct assessment of preschool children's science. A collaborative team from the University of Miami, the Miami-Dade County Head Start/Early Head Start program, and Children's Progress will create an Item Response Theory (IRT)-based, computer adaptive test of science, Lens on Science, primarily for use with low-income preschool children.

Project Activities:Lens on Science will be based on previous research by the University of Miami team that resulted in a flip-book style preschool science test, which demonstrated preliminary evidence of internal and external validity. The team will expand the current item bank and translate existing and new items to a computer platform. Lens on Science will undergo multiple phases of testing following the guidelines for measurement development from the Standards for Educational and Psychological Testing (AERA, APA, & NCME 1999). Multiple forms of validity evidence will be collected, such as response process evaluation, evaluation of estimation procedures, reliability analyses, and differential item functioning. While data will be collected with children served by Miami-Dade Head Start, the final version of Lens on Science is intended to be used nationally.

Products: The product of this project will be Lens on Science, a computer-administered, adaptive, IRT-based science summative assessment for preschool children with accompanying administration and technical manuals. The measure is intended to be used to assess children's knowledge of three science content areas (life science, earth/space science and physical/energy science), and children's science processing skills.

Structured Abstract

Purpose: The purpose of this study is to develop an IRT-based, computer adaptive test of preschoolers' science knowledge, Lens on Science, and to gather preliminary evidence of internal and external validity.

Setting: The study sample will include Head Start children residing in Miami-Dade County. This county includes a large urban city, surrounding incorporated suburbs, and southern rural farm areas. The Miami-Dade County Head Start program serves over 7,000 low-income children and families. For the past 5 years, science has been a targeted focus in selected Head Start centers in the county. However, given the size of the program and staff turnover, there is currently a wide range of expertise in science from novice to expert in the teacher workforce, and similar variability in the science abilities of the children who will participate in this project.

Population: Approximately 920 preschool-aged children residing in Miami-Dade County and attending Head Start programs will participate across the 4-year time period of the award.

Research Design and Methods:Lens on Science will be designed using the guidelines for measurement development from the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999). These guidelines include review of items by a panel of experts for content validation, collecting response-process validation data, collecting data in a non-adaptive framework to obtain estimates of the item parameters, evaluating the estimation procedures, and testing the fully adaptive computerized assessment system.

Control Condition: There is no control condition.

Key Measures: The research team will use data from the Galileo System for the Electronic Measurement of Learning, the Preschool Learning Behaviors Scale, the Devereux Early Childhood Assessment, and the Learning Express to examine the validity of the preschool science measure.

Data Analytic Strategy: The measurement framework used in the adaptive assessment will be the Rasch model for dichotomously scored items with Bayesian ability estimation and item selection algorithms. Administration and reporting of the test requires several statistical estimation procedures to compute parameters such as ability levels (using the Bayesian expected a-posteriori estimator), the standard error of the ability estimator (estimated using the standard deviation of the posterior distribution of ability), and mathematical algorithms for item selection (determined by the posterior-weighted item information function).