7
7 Combining the indicators Indicators are measured on different scales –Categorical (eg. Yes/No) –Proportional (eg. proportion of patients waiting longer than 15 months) –Rates (eg. mortality rate within 30 days following selected surgical procedures) Further complication –Performance on some indicators is measured against published targets – define thresholds –Performance on other indicators is based on relative differences between trusts

12
12 The old statistical method - problematic Not a proper statistical hypothesis test Differentiating between trusts based on differences that exceed levels of sampling variation On some indicators, this led to the assignment of too many NHS trust to the significantly good/ bad bands on some indicators

13
13 Working example - standardised readmission rate of patients within 28 days of initial discharge Significantly below average Below average AverageAbove average Significantly above average Total 326401349140

16
16 Z scores Standardised residual Z scores are used to summarise extremeness of the indicators Funnel plot limits approximate to the naïve Z score Naïve Z score given by –Z i = (y i –t)/s i –Where y i is the indicator value, and s i is the local standard error

17
17 Dealing with over-dispersion Three options were considered –Use of an interval null hypothesis –Allow for over-dispersion using a multiplicative variance model –…or a random-effects additive variance model

18
18 Interval null hypothesis Similar to the naïve Z score or standard funnel limits Uses a judgement of what constitutes a normal range for the indicator Define normal range (eg percentiles, national rate ± x%) Funnel limits then defined as: –Upper/ lower limit = Range limit ± (x * s i 0 ) Reduces number of significant results But might be considered somewhat arbitrary Interval could be defined based on previous years data, or prior knowledge Makes minimal use of the sampling error

20
20 Multiplicative variance model Inflates the variance associated with each observation by an over-dispersion factor ( ): – Z i 2 = Pearson X 2 – = X 2 / I Limits on funnel plot are then expanded by Do not want to be influenced by the outliers we are trying to identify Data are first winsorised (shrinks the extreme z- values in) Over dispersion factor could be provisionally defined based on previous years data Statistically respectable, based on a quasi- likelihood approach

23
23 Winsorising Winsorising consists of shrinking in the extreme Z- scores to some selected percentile, using the following method. 1.Rank cases according to their naive Z-scores. 2.Identify Zq and Z1-q, the (100*q)% most extreme top and bottom naive Z-scores, where q might, for example, be 0.1 3.Set the lowest (100*q)% of Z-scores to Zq, and the highest (100*q)% of Z-scores to Z1-q. These are the Winsorised statistics. This retains the same number of Z-scores but discounts the influence of outliers.

25
25 Random effects additive variance model Based on a technique developed for meta-analysis Originally designed for combining the results of disparate studies into the same effect In meta-analysis terms, consider the indicator value of each trust to be a separate study Essentially seeks to compare each trust to a null distribution instead of a point Assumes that E[y i ] = i, and V[ i ] = Uses a method-of-moments method to estimate (Dersimonian and Laird, 1986) Based on winsorised estimate of

30
30 Why we chose the additive variance method Generally avoids situations where two trusts which have the same value for the indicator get put in different bands because of precision A multiplicative model would increase the variance at some trusts more than at others – e.g. a small trust with large variance would be affected much more than a large trust with small variance By contrast, an additive model increases the variance at all trusts by the same amount Better conceptual fit with our understanding of the problem, that the factors inflating variance affect all trusts equally, so an additive model is preferable