Pooling and Meta-analysis. Tony O Hagan

Transcription

1 Pooling and Meta-analysis Tony O Hagan

2 Pooling Synthesising prior information from several experts 2

3 Multiple experts The case of multiple experts is important When elicitation is used to provide expert input to a decision problem with substantial consequences, we generally want to use the skill of as many experts as possible However, they are not separate pieces of evidence The experts typically base their judgements on the same or similar information It is better to treat them as different attempts to formulate the same information The prior information The question then arises of how to pool their judgements into a single prior density 3

4 Aggregating expert judgements Two approaches Aggregate the distributions Elicit a distribution from each expert separately Combine them using a suitable formula Called mathematical aggregation or pooling Aggregate the experts Get the experts together and elicit a single distribution Called behavioural aggregation Neither is without problems 4

6 Linear opinion pool The simplest way to combine the experts probability distributions is to average them 0.4 f 0 (x) = Σ i f i (x) / N Blue and green lines = two expert distributions Red line = pooled distribution y More generally, we can give weights w i to the experts x f 0 (x) = Σ i w i f i (x) / Σ i w i To reflect the experts different levels of expertise 6

7 Multiplicative opinion pool If instead we average the logarithms of the experts probability distributions this amounts to multiplying f 0 (x) = (Π i f i (x) ) 1/N Then scale the result so that it integrates to 1 Blue and green lines = two expert distributions Red line = pooled distribution More generally, we can give weights w i to the experts f 0 (x) = (Π i f i (x) w i ) 1/Σ iwi Then scale the result so that it integrates to 1 To reflect the experts different levels of expertise y x 7

8 Practical comparison Linear Each expert could be right Pooled distribution covers the range of their beliefs y 0.4 Multiplicative Both experts are right Pooled distribution based on the intersection of beliefs y x x 8

9 Theoretical comparison Linear Consistent when you simplify Pr 0 (X > 0) is average of Pr i (X > 0) But not when you add information f 0 (x X > 0) is not the average of f i (x X > 0) Multiplicative Consistent when you add information f 0 (x X > 0) is the logaverage of f i (x X > 0) But not when you simplify Pr 0 (X > 0) is not the logaverage of Pr i (X > 0) There is no way to pool without losing some desirable properties 9

10 Behavioural aggregation The alternative to mathematical aggregation Get the experts in a room together and ask them to elicit a single distribution Advantages Opportunity to share knowledge Avoids arbitrary choice of pooling rule Allows more subtle forms of aggregation y 0.4 y ? x x 10

11 But... More psychological hazards Group dynamic dominant/reticent experts Tendency to end up more confident Block votes Requires careful management What to do if they can t agree? End up with two or more composite distributions Need to apply mathematical pooling to these But this is rare in practice 11

12 Meta-analysis Synthesising indirect evidence 12

13 Basic normal meta-analysis The canonical context is to synthesise evidence from a set of clinical trials, all testing the same drug/treatment These are usually found from a systematic review of the literature In general, suppose we have found N trials Denote evidence from trial t by E t Could in principle be individual patient data More usually it comprises summary statistics Consider the case where E t = (z t, s t ) Where z t is estimated treatment effect and s t its standard error And we assume normally distributed estimate Lots of other situations arise We want to synthesise to make inference about true effect θ 13

14 A simple model Assume z t ~ N(θ, s t2 ) Meaning z t has a normal distribution with mean the true effect θ and with variance s t 2 These are now multiple data items as in Session 1 Combine with prior information Usually formulated as weak prior Derive posterior inferences about θ However, this may be too simple Experience shows trial estimates are often too varied Each trial has different features Recruitment criteria, trial conditions So each may be estimating a slightly different θ 14

16 Numerical example The data here are made up, but intended to represent a realistic scenario 9 trials of widely varying sizes (and hence standard errors) Trial z t s t If all the trials are providing unbiased estimates of the same θ then for instance the difference between trial 6 and trials 4 and 9 is surprising 16

17 Numerical example (contd.) The simple model produces a normal posterior distribution for θ with mean and standard deviation The random effects model produces a posterior distribution for θ with mean and standard deviation The posterior mean of τ 2 is 0.52 The two results are quite different The value of τ 2 indicates that treatment effect varies widely with patient group Even when random effects appear small they should not be ignored 0,05 0,045 0,04 0,035 0,03 0,025 0,02 0,015 0,01 0, ,5 0 0,5 1 1,5 2 17

18 Data gaps It is very common to find that data concern a slightly different situation from the one of interest We are interested in a parameter θ, but the data inform us about a different parameter α α is related to θ, but we need to link the two in order to use the data to learn about θ I call this a data gap Or data discrepancy In the case of random effects meta-analysis Each trial provides information about a different parameter α t We build the link with θ by writing α t = θ + ε t Together with a distribution for the discrepancy ε t 18

19 Data gaps (contd.) There is usually very little information about discrepancies In this case just 9 trials to learn about the discrepancy variance Sometimes there is essentially no data to fill data gaps Hence the name Expert elicitation is then the only feasible way to fill them We ll encounter similar gaps and discrepancies elsewhere in this course 19

21 Beyond simple treatment comparisons Basic meta-analysis combines data from several trials about a single treatment effect Usually a difference between the treatment of interest and a comparator An alternative treatment, placebo or standard care Often we don t find multiple trials all with the same comparator Some compared with placebo and some compared with active comparator Combination therapies Here are some simple examples of what is called Mixed Treatment Comparison 21

22 Different comparators We are interested in the effect of treatment A compared with treatment B We have some trials comparing A with C and some comparing B with C Model the effects in the second group as β + ε t Model the effects in the first group as θ + β + ε t Then θ represents the additional effect of A versus B Now suppose we also have a 3-arm trial comparing A, B and C Now need to model each arm of each trial separately For a treatment A arm we write the mean response as θ + β + η a + ε t Where η a is a random effect of arm a and ε t is a trial random effect as before 22

23 Combination therapies In cancer treatment, it is normal to give patients a combination of drugs Also found in other serious conditions E.g. patients who have had a stroke or heart attack We may want to estimate the effect of a combination which has never appeared yet in any trial Suppose in a treatment arm, patients receive drugs A, B and C Then the mean effect in that arm could be modelled as a sum of drug effects θ A + θ B + θ C plus random effects But this assumes independent effects, and some interaction effects may also be modelled Interactions in combinations that have not yet been tested represent a data gap for which there is no evidence 23

24 Introducing covariates These mixed treatment comparison models are particular cases of a more general meta-regression framework For each trial effect or treatment arm we may have some covariates that might explain trial differences Average patient age or disease severity Trial duration We can model the mean response on a treatment arm or the mean effect in a trial using conventional regression techniques E.g. θ + βs t + ε t Where s t is average severity in trial t and β is regression coefficient Mixed treatment comparisons involve 0-1 covariates 0 if treatment not used, 1 if it is used 24

25 Summary of session 2 Combining beliefs of several experts is a different kind of synthesis Mathematical aggregation is simple but involves an arbitrary choice of pooling rule Behavioural aggregation is potentially more powerful but also more complex SHELF system to help facilitators Meta-analysis is a powerful range of techniques for synthesising published data sources Widely used for synthesising evidence from clinical trials Including mixed treatment comparisons and meta-regression Can be adapted to many other contexts We frequently need to be aware of data gaps/discrepancies Gaps are often hard to fill and may require expert elicitation 25

Summary of Probability Mathematical Physics I Rules of Probability The probability of an event is called P(A), which is a positive number less than or equal to 1. The total probability for all possible

Simple Linear Regression Chapter 11 Rationale Frequently decision-making situations require modeling of relationships among business variables. For instance, the amount of sale of a product may be related

Math 62 Statistics Sample Exam Questions 1. (10) Explain the difference between the distribution of a population and the sampling distribution of a statistic, such as the mean, of a sample randomly selected

CHAPTER 40 When Does it Make Sense to Perform a Meta-Analysis? Introduction Are the studies similar enough to combine? Can I combine studies with different designs? How many studies are enough to carry

Gregory Carey, 1998 General Linear Model - 1 The General Linear Model: Theory 1.0 Introduction In the discussion of multiple regression, we used the following equation to express the linear model for a

and P (B X). In order to do find those conditional probabilities, we ll use Bayes formula. We can easily compute the reverse probabilities A short introduction to Bayesian statistics, part I Math 18, Mathematical

An Introduction to Bayesian Statistics Robert Weiss Department of Biostatistics UCLA School of Public Health robweiss@ucla.edu April 2011 Robert Weiss (UCLA) An Introduction to Bayesian Statistics UCLA

LESSON SEVEN CONFIDENCE INTERVALS FOR MEANS AND PROPORTIONS An interval estimate for μ of the form a margin of error would provide the user with a measure of the uncertainty associated with the point estimate.

Name: Class: Date: Confidence Intervals and Hypothesis Testing Multiple Choice Identify the choice that best completes the statement or answers the question. 1. The librarian at the Library of Congress

Lecture 13: Kolmogorov Smirnov Test & Power of Tests S. Massa, Department of Statistics, University of Oxford 2 February 2016 An example Suppose you are given the following 100 observations. -0.16-0.68-0.32-0.85

Bayesian Sample Size Calculations Tony O Hagan, University of Sheffield John Stevens, AstraZeneca R&D Charnwood The Problem A trial is to be designed comparing Treatment 1 with Treatment 2 Both efficacy

Statistics for Engineers 4-1 4. Introduction to Statistics Descriptive Statistics Types of data A variate or random variable is a quantity or attribute whose value may vary from one unit of investigation

Correlation Regression Bivariate Normal Suppose that X and Y are r.v. s with joint density f(x y) and suppose that the means of X and Y are respectively µ 1 µ 2 and the variances are 1 2. Definition The

Lecture 5: Hypothesis Testing What we know now: OLS is not only unbiased it is also the most precise (efficient) unbiased estimation technique - ie the estimator has the smallest variance (if the Gauss-Markov

Wording of Final Conclusion Slide 1 8.3: Assumptions for Testing Slide 2 Claims About Population Means 1) The sample is a simple random sample. 2) The value of the population standard deviation σ is known

Probability and Statistics Lecture 9: 1 and -Sample Estimation to accompany Probability and Statistics for Engineers and Scientists Fatih Cavdur Introduction A statistic θ is said to be an unbiased estimator

Ismor Fischer, 5/9/01 5.-1 5. Formal Statement and Examples Comments: Sampling Distribution of a Normal Variable Given a random variable. Suppose that the population distribution of is known to be normal,

Systematic Reviews and Meta-analyses Introduction A systematic review (also called an overview) attempts to summarize the scientific evidence related to treatment, causation, diagnosis, or prognosis of

6. Duality between confidence intervals and statistical tests Suppose we carry out the following test at a significance level of 100α%. H 0 :µ = µ 0 H A :µ µ 0 Then we reject H 0 if and only if µ 0 does

Chapter 855 Introduction Linear regression is a commonly used procedure in statistical analysis. One of the main objectives in linear regression analysis is to test hypotheses about the slope (sometimes

Practice for Chapter 9 and 10 The acutal exam differs. SHORT ANSWER. Write the word or phrase that best completes each statement or answers the question. Find the number of successes x suggested by the

Comparing Two Populations OPRE 6301 Introduction... In many applications, we are interested in hypotheses concerning differences between the means of two populations. For example, we may wish to decide

Sampling & Confidence Intervals Mark Lunt Arthritis Research UK Centre for Excellence in Epidemiology University of Manchester 18/10/2016 Principles of Sampling Often, it is not practical to measure every

Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem

Least Squares Introduction We have mentioned that one should not always conclude that because two variables are correlated that one variable is causing the other to behave a certain way. However, sometimes

CS 650: Computer Vision Bryan S. Morse BYU Computer Science Statistical Basis Training: Class-Conditional Probabilities Suppose that we measure features for a large training set taken from class ω i. Each

LECTURE 5 Hypothesis Testing in the Classical Regression Model The Normal Distribution and the Sampling Distributions It is often appropriate to assume that the elements of the disturbance vector ε within

Introduction to Hypothesis Testing This deals with an issue highly similar to what we did in the previous chapter. In that chapter we used sample information to make inferences about the range of possibilities

Estimation of the Mean Variance Portfolio Model In the mean variance framework, the optimal portfolio weight vector, x, is a function of the investor s preference parameters, c, and the first two moments

Transferability of Economic Evaluations in Clinical Trials Henry Glick Institutt for helseledelse og helseøkonomi November 25, 2008 The Problem Multicenter and multinational studies are the norm for the

Overview: The SMEEACT (Software for More Efficient, Ethical, and Affordable Clinical Trials) web interface (http://research.mdacc.tmc.edu/smeeactweb) implements a single analysis of a two-armed trial comparing

Chapter 1 Introduction to Econometrics Econometrics deals with the measurement of economic relationships. It is an integration of economics, mathematical economics and statistics with an objective to provide

Regression Analysis Prof. Soumen Maity Department of Mathematics Indian Institute of Technology, Kharagpur Lecture - 2 Simple Linear Regression Hi, this is my second lecture in module one and on simple

HYPOTHESIS TESTING (TWO SAMPLE) - CHAPTER 8 1 PREVIOUSLY estimation how can a sample be used to estimate the unknown parameters of a population use confidence intervals around point estimates of central

C A S I O f x - 8 2 T L UNIVERSITY OF SOUTHERN QUEENSLAND The Learning Centre Learning and Teaching Support Unit MASTERING THE CALCULATOR USING THE CASIO fx-82tl Learning and Teaching Support Unit (LTSU)

CSS.com Chapter 35 Standard Deviation Calculator Introduction The is a tool to calculate the standard deviation from the data, the standard error, the range, percentiles, the COV, confidence limits, or

This paper explains the research conducted by Minitab statisticians to develop the methods and data checks used in the Assistant in Minitab 17 Statistical Software. The simple regression procedure in the