slide deck

Instrument Development and Psychometric
Evaluation: Scientific Standards
May 2012
Dynamic Tools to Measure Health Outcomes from the Patient Perspective
PROMIS® Standards document
• Patient-Reported Outcome Measurement
Information System (PROMIS®) provides
clinicians and researchers access to efficient,
valid and responsive self-reported measures in
health, including symptoms, function and wellClick to edit Master title style
being.
• Instrument Development and Psychometric
Evaluation: Scientific Standards describes a set
of standards that serve as
• the scientific foundation and guidance for the development of
PROMIS
• evaluation of PROMIS item banks and instruments.
PROMIS® Standards document
• Practices are based on measurement science,
experience of PROMIS® investigators, published
literature on methodology of PROMIS ® and
other PRO measurement development
• Standards are operationalized by series of
Click tothat
editprovide
Masterdetailed
title style
guidelines
guidance for
item bank development and psychometric
evaluation
PROMIS® Scientific Standards
Address:
1)
2)
3)
4)
5)
6)
7)
8)
9)
Defining target concept and conceptual model
Generating and design of individual items
Constructing item pool
Determining item bank properties
Field testing and instrument formats
Click to edit Master title style
Validity
Reliability
Interpretability
Language translation and cultural adaptation
1.Defining the Target Concept and Conceptual Model
 Conceptual
model incorporating target
concept(s) should be defined and based on
extant literature with input from patients,
content and measurement experts,
clinicians, end users and stakeholders
Click to edit Master title style
• Placement of the instrument within
PROMIS® framework should be defined
2. Composing Individual Items
• Individual items should be refined through
cognitive interviewing to ensure:
–The meaning is understood as intended
–The item is clear and contains one concept
• Items also should be reviewed for:
Click to edit
–Translatability
–Literacy
–Readability
Master title style
2. Composing Individual Items, cont’d.
• Level of life course and cultural
harmonization should be addressed
• Existing PROMIS® item formats should be
considered and utilized as appropriate
to editused
Master
title style
• TheClick
language
should
be clear to
patients and consistent with results of
formative research
3. Constructing Item Pool
• Item pool should cover a full breath of
target construct
• Consideration should be paid to overlap of
facets already covered by extant PROMIS®
domains
Click to edit Master title style
• Consideration should be paid to
intellectual property issues
4. Determining Item Bank Properties
• Psychometric characteristics of items within an item
bank should be determined based on a representative
sample of respondents drawn from a relevant and
interpretable population
• Item banks should have good measurement
characteristics:
–Well-characterized and modeled dimensionality
Click
toofedit
Master
style error
–High
degree
information
andtitle
low standard
–Model fit, item and scale properties
• Differential item functioning (DIF) should be assessed
for key groups and its impact on measurement properties
identified
5. Testing and Instrument Formats
• Instrument formats should be appropriately
defined based on intended use and item bank
properties including:
–CATS
–Fixed length short-forms
–Profiles
–Screeners
• Adequate
scale
properties
and style
performance
Click
to
edit
Master
title
should be demonstrated and include assessment
of respondent burden
• Instruments that use different modes and
methods of administration should demonstrate:
–comparability of scale properties and performance
–assessment of respondent burden for each mode
6. Validity
• Validity for construct, content and criterion
should be addressed relative to a priori
hypothesized relationships with related
measures
– Description of methods and sample used to
evaluate validity, including hypotheses tested and
rationale
for edit
choice
of “gold
standard”
and
Click
to
Master
title
style
measures should be provided
• Final instrument should be re-reviewed by
experts and end-users to assess consistency
with or identify differences between original
definitions and final product
6. Validity, cont’d.
• If an instrument is purported to be
responsive, relevant anchor-based methods in
representative populations should be provided
• Longitudinal data collected will compare a
group expected to change with a group
expected to remain stable
Click
to
edit
Master
title
style
• Rationale should be provided for the external
anchors used to document change
•Rationale should be provided for the time
intervals used for assessment
7. Reliability
• Reliability of an instrument should be
described, including methods used to collect
data and estimate reliability
• Internal consistency reliability estimates may
consist of:
–Information and standard errors at different
Click to
Master
styletheory)
locations
of edit
the scale
(itemtitle
response
–Reliability estimates and standard errors for all
score elements (classical test theory)
7. Reliability, cont’d.
• Reproducibility of the measure should be
described, providing
–Rationale to support the design of the study
–The interval between initial and subsequent
administration
supporttitle
the style
assumption that
Click to edittoMaster
the population is stable
8. Interpretability
• The degree to which one can assign easily
understood meaning to the instrument’s
quantitative scores should be described
• Rationale should be provided for the
external anchors used to facilitate
Click to editofMaster
interpretability
scorestitle style
• Information should be provided on the
ways in which data from an instrument
should be reported or displayed
8. Interpretability, cont’d.
• Availability of comparative data from the
general population and/or group-specific
scores should be described
• Guidance should be provided on the
meaningfulness
of scores
changes in
Click to edit Master
titleand
style
scores for use by researchers and clinicians
9. Translation and Cultural Adaptation
• Translation of items and instruments
should include both forward and backward
translations of all items, response choices,
and instructions
•Translation of items, response choices,
to edit Master
style
and Click
instructions
shouldtitle
be obtained
through
–Iterative process of forward and back
translation
–Bilingual expert review
–Pre-testing with cognitive debriefing
9. Translation and Cultural Adaptation, cont’d
• Harmonization across all languages and a
universal approach to translation should
guide the process
Click to edit Master title style
Appendix
• Each standard refers to guideline documents
in the appendix for further description of
processes for performing recommended
practices
• Appendix also includes the PROMIS®
Instrument Maturity Model that describes the
Click
to edit Master
title stylefrom
stages
of instrument
development
conceptualization through evidence of
psychometric properties in multiple diverse
populations.
PROMIS Instrument Maturity Model Stages
PROMIS Instrument Maturity Model Stages
Stage Descriptions
1
1A
2
1B
2A
3
2B
3A
4
3B
4A
5
4B
5
Item Pool
Preliminary Item
Bank
Calibrated Item
Bank
Item Bank, Profile or
Instruments
Global Health Measure Validated
- Prelim Validated
Instruments Maturing
Prelim responsive- Instruments
ness
Item Bank Expansion
Conceptualized
Ready for
Calibration
Dimension-ality
Assessed &
Calibrated
Validity (Construct &
Concurrent - limited)
Prelim responsive- Extensive
ness
validity &
responsiveness
in general and
pertinent
population
samples
Item bank
How scores can be used to understand and
modifications respond to health care needs and differences
population specific or in health is determined and documented
expansion/ refinement
CrossSectional,
Population
Specific
Fully Mature Instruments:
Score Interpretations
Click to edit Master title style
Internal Psychometric Criteria
Criteria
Applies To
QUALITATIVE: Conceptual documentation and evidence supporting content validity
All stages
Item response theory (IRT): Dimensionality Specified; Item calibration; information and DIF analyses
All but stage 1A
Classical test theory (CTT): Evidence supporting dimensionality, reliability and validity (e.g. concurrent validity
with legacy)
All but stage 1A
POPULATION: Sample variability reflects variability in construct
Stages 2, 3, and 4
FORMAT: CAT and short form measures; Computer, paper forms
Stages 3A, 3B, and
4A
Continued Documentation of Relevance of Item Content and Generalizability as needed
Stages 1 and 2
Continued Documentation of Relevance of Item Content and Generalizability
Stages 4B and 5
External Psychometric Criteria
Criteria
Applies To
IRT: DIF analyses by different disease conditions and relevant population characteristics (e.g. age, sex, etc.)
Stages 3 and 4
CTT: Evidence supporting responsiveness and interpretation guidelines (MID, responder criteria)
Stages 3 and 4
POPULATION: General population and multiple disease conditions; Relevant language translations
Stages 3 and 4
FORMAT: CAT, short form, and study tailored forms
Stages 3 and 4
MODE: Evidence supporting multiple modes of administration (CAT, paper, IVRS, computer)
Stages 3 and 4