Davy, Zowie and Quinn, Casey and Wilson, Helen and Togher, Fiona and Siriwardena, A Niroshan
(2011)
Quantitative versus qualitative notions of validity: investigating the validity of PSYCHLOPS as a patient outcome measure in trials of insomnia and sleep problems.
In: 40th Annual Scientific Meeting of the SAPC, 6-8 July 2011, University of Bristol, Bristol .

Abstract

IntroductionPSYCHLOPS (‘Psychological Outcome Profiles’)is a novel patient-reported outcome measure developed by Ashworth and colleagues that allows clients to evaluate their progress by measuring severity scores for self-defined problems at the start of, during and after therapy. PSYCHLOPS was administered in a study, ‘Resources for Effective Sleep Treatment (REST)’, designed to assess the feasibility of training primary care clinicians (GPs and nurses) to deliver ‘problem focused therapy’ for insomnia to improve sleep outcomes in adults with insomnia. We aimed in this sub-study to assess both qualitative and quantitative notions of validity of PSYCHLOPS in sleep studies.MethodsPSYCHLOPS was administered by a practice nurse to trial participants in two intervention primary care sites (participants offered ‘problem focused therapy’) and two control sites (participants offered sleep hygiene information) as part of a cluster randomized feasibility study. Other predetermined outcome measures including the Pittsburgh Sleep Quality Index (PSQI), Insomnia Severity Index (ISI), Beck Depression Inventory (BDI) and sleep diaries. We investigated qualitative and quantitative markers of validity including content validity (nature of response to items), internal validity (reliability), criterion validity (strength of relationship with a related variable) and construct validity (strength of relationship with an underlying variable). ResultsA qualitative analysis of test content showed a representative sample of the behavior domain expected in relation to sleep problems including sleep, sleep-related problems and underlying psychosocial and physical problems such as anxiety, worry and arthritis. A positive, statistically significant correlation was found between PSYCHLOPS and ISI (Kendall’s tau = 0.47, p<0.001) but not between PSYCHLOPS and PSQI (Kendall’s tau = 0.13, p=0.24) suggesting partial criterion validity with regards to insomnia impact. We also found a positive correlation between PSYCHLOPS and BDI and (Kendall’s tau = 0.47, p<0.001) indicating construct validity.ConclusionPSYCHLOPS demonstrated some aspects of quantitative validity supporting its wider use in this study setting. However, applying qualitative notions of validity uncovered a number of assumptions and response biases that should be acknowledged in future studies using the tool. Qualitative analysis added an additional dimension to the assessment of validity which should be considered in other validation studies.

Item Type:

Conference or Workshop Item (Presentation)

Additional Information:

Introduction
PSYCHLOPS (‘Psychological Outcome Profiles’)is a novel patient-reported outcome measure developed by Ashworth and colleagues that allows clients to evaluate their progress by measuring severity scores for self-defined problems at the start of, during and after therapy. PSYCHLOPS was administered in a study, ‘Resources for Effective Sleep Treatment (REST)’, designed to assess the feasibility of training primary care clinicians (GPs and nurses) to deliver ‘problem focused therapy’ for insomnia to improve sleep outcomes in adults with insomnia. We aimed in this sub-study to assess both qualitative and quantitative notions of validity of PSYCHLOPS in sleep studies.
Methods
PSYCHLOPS was administered by a practice nurse to trial participants in two intervention primary care sites (participants offered ‘problem focused therapy’) and two control sites (participants offered sleep hygiene information) as part of a cluster randomized feasibility study. Other predetermined outcome measures including the Pittsburgh Sleep Quality Index (PSQI), Insomnia Severity Index (ISI), Beck Depression Inventory (BDI) and sleep diaries. We investigated qualitative and quantitative markers of validity including content validity (nature of response to items), internal validity (reliability), criterion validity (strength of relationship with a related variable) and construct validity (strength of relationship with an underlying variable).
Results
A qualitative analysis of test content showed a representative sample of the behavior domain expected in relation to sleep problems including sleep, sleep-related problems and underlying psychosocial and physical problems such as anxiety, worry and arthritis. A positive, statistically significant correlation was found between PSYCHLOPS and ISI (Kendall’s tau = 0.47, p<0.001) but not between PSYCHLOPS and PSQI (Kendall’s tau = 0.13, p=0.24) suggesting partial criterion validity with regards to insomnia impact. We also found a positive correlation between PSYCHLOPS and BDI and (Kendall’s tau = 0.47, p<0.001) indicating construct validity.
Conclusion
PSYCHLOPS demonstrated some aspects of quantitative validity supporting its wider use in this study setting. However, applying qualitative notions of validity uncovered a number of assumptions and response biases that should be acknowledged in future studies using the tool. Qualitative analysis added an additional dimension to the assessment of validity which should be considered in other validation studies.