Delphi Study summary.

How are health care treatments developed?

To help patients, doctors and other health professionals make decisions about treatments, we need evidence about what works best. Treatments are developed and tested by researchers to make sure they work and are safe. To do this researchers need to look at the effects those treatments have on patients. Researchers do this by measuring an ‘outcome’. For example, in a study of how well a new asthma treatment works, ‘outcomes’ might include:

A measure of how fast you can blow air out of your lungs

Night time wheeze

Asthma quality of life measure

How do researchers decide on what outcomes are important to measure in research studies?

When researchers design research studies to investigate treatments for health conditions they need to measure outcomes that are important and relevant to those people affected by the condition. To decide which outcomes are important researchers need to get everyone's opinion and try to reach agreement, or “consensus”, on the most important outcomes. In order to do this, researchers carry out a 'consensus exercise’. One way of doing this is by using something called a 'Delphi' study. In a Delphi study researchers identify groups of people who are “experts” in the health condition they are interested in. “Experts” are:

People with personal experience of the condition, for example, patients, carers and service users (it doesn’t matter how long the person has had the condition for, their opinion is incredibly valuable).

Health professionals with expertise in treating and caring for people with the condition.

Participants taking part in a Delphi study are asked to give their opinion on what outcomes are most important. The study is anonymous to make sure everyone has an equal say.

What happens early on in a Delphi Study?

The research team will have developed a long list of possible outcomes that they want to ask the experts about. This list is likely to have been created after looking at research papers, and sometimes after interviewing patients (see the flowchart on page 2)

What happens next?

Each participant is usually sent the list in the form of a questionnaire / survey by email and asked to score the importance of each outcome. If, in their opinion, there are key outcomes missing from the list, they are encouraged to add these to the list. We refer to this as “Round 1” of the Delphi study.

Each participant sends their ratings back to the research team, who then summarise the responses from the group as a whole and send this summary back to each participant in what we refer to as Round 2 of the Delphi process. At this stage each participant is given a reminder of how they scored the outcome last time and the range of scores of the rest of the group.

No-one in the group can see another individual’s scores; they can only see the overall results for the group as a whole. Using this information each participant is asked to reflect on their own view and on the view of the group and to decide whether to stick with their original rating or change it. Through the whole process no-one is under any pressure to change their rating if they don’t want to. It is perfectly fine for people to stick with their original rating even if they rated the outcome differently to the rest of the group.

The responses of the participants are then sent back again to the research team who again collate the information. Every time the researchers ask the participants for their opinions we call this a 'round' of the Delphi. Studies differ in how many rounds takeplace, but usually this is two or three times. Each time the idea is that the participants review their previous score based on what the group rated in their last round.

Usually at the end of the questionnaires / survey the researchers invite all the experts to get together face to face to discuss the results, although people do not have to attend if they do not want to. At the end of this process the research team produce a report on what the experts have agreed as the most important outcomes. These are called the ‘core outcomes’ for a particular health condition.

This leaflet has been modified from the COMET website (http://comet-initiative.org). COMET would like to thank the many patients, parents and carers who gave their time to comment on the development of this summary.

What are core outcome sets?

HOWHEALTH CARE TREATMENTS DEVELOPED?

To help patients, doctors and other health professionals make decisions about treatments, we need evidence about what works best. Treatments are developed and tested by researchers to make sure they work and are safe. To do this researchers need to look at the effects those treatments have on patients. Researchers do this by measuring an ‘outcome’. For example, in a study of how well a new asthma treatment works, ‘outcomes’ might include:

A measure of how fast you can blow air out of your lungs

Night time wheeze

Asthma quality of life measure

WHAT ARE THE CHALLENGES IN MEASURING OUTCOMES?

At the moment, different studies looking at treatments for the same condition often measure different outcomes. For instance, imagine two studies of how to treat migraine.

Study A - researchers measure days off sick as an outcome

Study B - researchers measure symptoms of pain as an outcome

When the two studies are finished, we cannot compare or combine their results because they have used different outcomes. We would not be comparing like with like.

Another problem is when a study team set out to collect information on several outcomes but in the end decide to publish the results on only some of the outcomes. This sometimes happens when the findings for particular outcomes differ from those the team hoped for.

WHY IS THIS A PROBLEM?

Other researchers could struggle to work out exactly how effective a treatment is because:

They don’t have full results from the study

The information they do have might be biased

HOW CAN WE SOLVE THIS PROBLEM?

If all studies in a particular health condition used the same outcomes, they could all be compared and combined. This would reduce waste by making best use of all the research. When a set of main outcomes has been agreed for a health condition, it’s called a ‘core outcome set’. If all studies in a particular condition, such as migraine, then measured and reported all these core outcomes, we could:

Bring together all the studies to get a better understanding of which treatments are best

Avoid the problem of some studies only reporting a selection of the outcomes that were measured.

HOW ARE CORE OUTCOMES AGREED UPON?

Deciding which outcomes should be core requires a great deal of discussion. Core outcomes have to be relevant to patients, carers and health professionals. People working on core outcome sets need to make sure that this expertise – from patients, carers and professionals – is used to agree on the core outcomes. To do this they often use ‘consensus methods’.

WHAT ARE CONSENSUS METHODS?

Consensus methods are surveys (Delphi study), meetings and discussions where the opinions of relevant experts are drawn together. Sadly, patients and carers have not always been involved in this type of work, but now they are being included recognising the importance of their first-hand experience of living with a condition.

WHY IS IT SO IMPORTANT TO INVOLVE PATIENTS?

Core outcome sets need to include outcomes that are most relevant to patients and carers, and the best way to do this is to involve patients and carers in their development. There are examples of how involving patients identified an outcome that was important to them as a group but which might have been overlooked if practitioners had done the work on their own.

HOW ARE CORE OUTCOME SETS USED WHEN THEY HAVE BEEN AGREED?

When a core outcome set has been agreed the hope is that researchers will use it in all studies for a particular condition, adding in other outcomes if they wish. For example if every migraine study used the same core outcome set, their findings could be compared and combined correctly. In the long run, this will improve the quality of information about which treatments work and which don’t and help people make better choices.

WHAT IS COMET AND HOW IS IT HELPING?

COMET stands for the ‘Core Outcome Measures in Effectiveness Trials’ Initiative. It involves people from around the world and in many different areas of health and social care and was set up to help in two main ways:

to provide the COMET database as a central point where researchers, practitioners and patients can find core outcome sets that have already been developed or are still under development.

Provide materials to support the teams who are working on core outcome sets

COMET is funded by the UK Medical Research Council and the European Union and has links with researchers across the world. This information leaflet is a modified copy from the COMET website (http://comet-initiative.org)

If you would like further information about COMET please contact Liz Gargon (COMET Project Co-ordinator) gargon01@liverpool.ac.uk or Heather Bagley (COMET Patient and Public Involvement Co-ordinator, heather.bagley@liverpool.ac.uk). Further information on COMET can be found at www.comet-initiative.org.