Quality Appraisal

The full texts of all selected articles were retrieved; this included the full texts of the initial systematic reviews and the primary studies. Where a cost-effectiveness study employed estimates of effectiveness from a separate trial or meta-analysis, the full text of the primary effectiveness study was also retrieved for QA purposes.

A quality assessment and scoring system was developed based on recognised standards in systematic reviewing, reporting of randomised trials and economic studies from the guidance in QUOROM (Moher et al, 2009), AGREE (AGREE, 2003), and the CONSORT statements.

Four quality appraisal tools were used to score different study types:

meta-analyses

trials / RCTs

economic studies (CBAs, CEAs)

econometric studies (modelling)

Where effect data came from a review (meta-analysis), QA scores were calculated for the review using (1). Where effect data comes from primary studies, QA scores were calculated using (2)-(4). Some primary studies that report both effectiveness and CE data (e.g. a CBA based on an RCT) were scored using both (2) and (3) and the scores reported separately.

Each tool has a number of questions answered yes / no / don’t know. ‘Yes’ scored 1, ‘no’ and ‘don’t know’ score 0. The total score was converted into a rating of ++, + or –.

REA Quality Assurance questions

Study type

Questions

Scoring and grade

Reviews/meta-analysis

Does the review state the search strategies and inclusion criteria used?
Does the review report quantitative outcome data for each study?
Does the review include an assessment of study quality?

3 ++
2 +
1 -

Trials/RCTs

Is the follow-up period = 6 months from baseline?
Is the analysis performed on an intention to treat basis?
Is the attrition rate < 25%?
Are the treatment and control groups similar or are any differences in the treatment and control groups controlled for using statistical or matching techniques?
Is the allocation to the treatment and control groups random?

4-5 ++
3 +
0-2 -

Economic studies

Are all the major costs of the intervention included in the analysis?
Are costs calculated using market values?
Is the perspective adopted by the analysis clear?
Is a sensitivity analysis undertaken?
Does the study report the base year adopted by the analysis?
Is effectiveness data used in the modelling from a meta-analysis or an RCT?

4-6 ++
3 +
0-2 -

Econometric / modelling studies

Have the statistical assumptions (of the statistical methods/models used) been reported as being met?
Is there an adequate sample size for the analysis used?
Do the authors describe strategies to minimise bias and confounding?
In studies conducted at one point in time, is there an unambiguous direction of causality?