Pre-Conference Introductory Workshop

Wednesday, June 23, 1:00-5:00 pm

This introductory workshop began with a discussion of the basics of item response theory (IRT) modeling, including
discussions of defining model parameters, model assumptions, item and test characteristic and information curves,
frequently-used IRT models, and differences between traditional and IRT methods. The course discussed the role
of IRT modeling in health outcomes research of evaluating and developing questionnaires, identifying differential
item functioning, linking instruments, developing item banks, and assessing change. Critical issues for applying
this methodology in health outcomes research was discussed.

This session reviewed the current state of the science in health outcomes measurement. What defines a "quality"
or psychometrically-strong measure? What are the limitations of the instruments we use today? Are we satisfied
with the analytical tools used to evaluate our measures?

9:20 am

The Traditional and Modern Approaches to Outcomes Measurement

Ronald K. Hambleton, PhD
School of Education, University of Massachusetts at Amherst

This session defined and contrasted both the traditional approach (i.e., CTT) and the modern measurement (i.e.,
IRT) approach for data analysis and scoring. Also, it discussed the limitations and strengths of these methods
in health outcomes measurement.

10:10 am

Refreshment Break

10:40 am

Building and Revising Outcome Measures: Evaluating Item and Scale Functioning with the IRT model

This session discussed how IRT may be used to evaluate both the properties of the items and the scale and to
inform questionnaire revisions. What added information do we learn about the data from IRT modeling?

John A. Fleishman, PhD
Senior Social Scientist, Center for Financing Access and Cost Trends
Agency for Healthcare Research and Quality (AHRQ)

This session discussed the importance for evaluating DIF for determining measurement equivalence of item content
across instruments administered to different populations (differing by race, age, gender, etc.) or instruments
administered in different languages. How will DIF assessment benefit health disparities research?

Neil J. Dorans, PhD
Principal Measurement Statistician in the Research & Development Division
Educational Testing Service

This presentation addressed several issues associated with the linking of health outcomes. What is meant by
outcomes linking and equating? How does equating differ from other types of linking? What are common data collection
designs used to capture data for outcomes linking. What are some of the standard statistical procedures used
to link outcomes directly? What assumptions do they make? What role does IRT play in linking outcomes? What
assumptions do IRT methods make? Illustrations of different kinds of linkages were provided.

David Thissen, PhD
Professor, Quantitative Program, Department of Psychology
University of North Carolina at Chapel Hill

This session discussed the critical role that IRT plays to develop the item bank and why CTT methods are inappropriate
for such a task. What challenges/decisions will we have to make in choosing both the appropriate IRT model and
the constructs we want to measure? How can we balance the assumptions of the unidimensional IRT models with
the multifaceted nature of the constructs measured in health outcomes research?

4:45 PM

Reception and Poster Session

Day Two (Friday, June 25)

8:00 am

The IRT Data Analysis Project: In parallel sessions, two research teams demonstrated the strengths and
challenges of applying IRT modeling for evaluating data in health outcomes research and behavioral science.
The sessions provided detailed examples of the logic and flow of analyses, the tackling of technical problems,
and the interpretation of results. Examples of input and output files along with analysis summaries were provided
to the audience.

This session discussed how computerized-adaptive testing (CAT) will revolutionize the way we measure outcomes
through the development of tailored short forms and computerized-adaptive tests. The speakers presented results
from their own work and discussed how item banking (short forms and CATs) will benefit the measurement of patient-reported
outcomes in observational studies, clinical trials, and clinical practice.

12:00 PM

Luncheon

1:15 PM

Critical Issues for Developing and Maintaining Item Banks and CATs
Bringing the Concept into Reality: The Idea of a National Item Bank

This session discussed the feasibility for developing and implementing item banks and CATs in observational
and clinical settings. What critical issues should we address and how can we overcome the barriers to issues
like proprietary rights? What are the costs and amount of work that are required to create these products? What
role should government (e.g., NIH, FDA) and private entities play in this effort?

Peter M. Fayers, PhD
Professor of Medical Statistics
University of Aberdeen Medical School

For both health outcomes and behavioral researchers, speakers made recommendations for how their field of study
will benefit by incorporating the methods discussed over the past two days. What demonstrations are necessary
to explore the potential benefits of item banking and CATs to health outcomes measurement? What role should
academia, industry, and government play to push this arena forward?