Quality Grading

Estimates of each of the four criteria used to evaluate interventions were graded for the quality of the methods and data employed. The quality grading scales were developed using existing scales in the fields of research design (NICE, 2008), economic evaluation (Drummond and Jefferson, 1996), and statistics (Lyon, 2008).

Cost effectiveness

The four estimates used to construct the cost per QALY gained estimates were graded separately: cost; effect; the relationship between effect and QALY gains; and the relationship between effect and healthcare cost savings. Each estimate was given a score between 0 and 6, which were aggregated to produce an overall score for cost per QALY gained and net cost per QALY gained estimates. Figure 1 below summarises the process used to translate scores into a quality grading.

Figure 1: Cost effectiveness quality grading

Key to grading

Confidence in the cost per QALY gained estimate

Score

Grade

High confidence

13 - 18

***

Medium confidence

6 - 12

**

Low confidence

0 - 5

*

Confidence in the net cost per QALY gained estimate

Score

Grade

High confidence

17 - 24

***

Medium confidence

8 - 16

**

Low confidence

0 - 7

*

The separate scores for each of the components of the cost-effectiveness grading system were constructed from the responses to the questions summarised in Figure 2.

Figure 2: Components of the cost-effectiveness quality grading

Reach

The data used to estimate the reach of an intervention were assessed for their accuracy and timeliness. A number of questions were asked about the data to produce scores for accuracy and timeliness, which were then combined into an overall score. The grading is summarised in Figure 3.

Figure 3: Reach quality grading

Key to grading

Confidence in the reach estimate

Score

Grade

High confidence

9 - 12

***

Medium confidence

4 - 8

**

Low confidence

0 - 3

*

Figure 4: Components of the reach quality grading

Inequality Score

The data used to estimate the inequality score from an intervention were assessed for their accuracy and timeliness. A number of questions were asked about the data to produce scores for accuracy and timeliness, which were then combined into an overall score. The grading is summarised in Figure 5.

Figure 5: Inequality score quality grading

Key to grading

Confidence in the inequality score estimate

Score

Grade

High confidence

6 - 12

***

Medium confidence

2 - 6

**

Low confidence

0 - 1

*

The separate scores for each of the components of the inequality score grading system were constructed from the responses to the methods questions summarised in Figure 6.