Wednesday, August 21, 2013

The Delphi technique is ‘a method for structuring a group communication process so that
the process is effective in allowing a group of individuals, as a whole, to deal with a
complex problem’ (Linstone and Turoff 1975:3). Furthermore, it is ‘a method for the
systematic solicitation and collation of judgments on a particular topic through a set of
carefully designed sequential questionnaires interspersed with summarized information and
feedback of opinions derived from earlier responses’ (Delbecq et al. 1975:10). It is used
most frequently to integrate the judgments of a group of experts. A key feature of this
technique, however, is that the respondents do not meet and their responses may be
anonymous. We still consider it to be a dialogue method, however, because ‘conversation’
between the parties occurs, even though it is not face-to-face.
Three separate groups of actors are generally involved:

Respondent group: those whose judgments are obtained through completing the process

Staff group: those who design the initial questions, summarise the responses and
prepare the questions for subsequent phases

Decision-makers: those wishing to receive a product such as a consensus position
from experts, or a recommendation (adapted from Delbecq et al. 1975).

Although some flexibility exists in implementation, the core method, as described by
Delbecq et al. (1975:11), is as follows:

First, the staff team in collaboration with decision makers
develops an initial questionnaire and distributes it…to the respondent group. The
respondents independently generate their ideas in answer to the first questionnaire and
return it. The staff team then summarizes the responses to the first questionnaire and
develops a feedback report along with the second set of questionnaires for the respondent
group. Having received the feedback report, the respondents independently evaluate earlier
responses. Respondents are asked to independently vote on priority ideas included in the
second questionnaire and mail their responses back to the staff team. The staff team then
develops a final summary and feedback report to the respondent group and decision
makers.

Variations of this basic approach include:

whether the respondent group is anonymous

whether open-ended or structured questions are used to obtain
information from the respondent group

whether the responses are collected in written form or
verbally—for example, over the phone

how many iterations of questionnaires and feedback reports are
used

what decision rules are used to aggregate the judgments of the
respondent group.

The number of participants can range from a few to many hundreds. The larger the number
of iterations employed, the closer to consensus will be the result. Written questionnaires
can be in pencil-and-paper form or distributed and returned using electronic communication
tools including email and the internet. Computer-based systems, using highly structured
questionnaires, can produce real-time findings.

Examples of its use in research integration

1. The environment: developing an environmental plan for a university

What was the context for the integration, what was the integration aiming to
achieve and who was intended to benefit?
Senior administrators at Dalhousie University in Halifax, Nova Scotia, Canada, were
aware of a significant gap between the university’s environmental policies and their
implementation. As a result, they resolved to develop an implementation plan that would be
acceptable to all those who would be responsible for making it work.
Those responsible for developing the implementation plan used the Delphi technique

to consult with key representatives of the university community in
order to generate ideas about the most desirable and feasible ways in which to
incorporate the new Environmental Policy into the activities and structure of the
university…Modifying a Delphi study for policy research can be used to generate ideas
and provide decision-makers with the strongest arguments for and against different
resolutions to an issue. (Wright 2006:763)

Who did the integration, how was it undertaken and what was
integrated?
A panel of 28 individuals was selected, with equal numbers drawn from the identified
key stakeholders: ‘students, staff, faculty, and administrators.’ A core feature was that
the Delphi study participants would be anonymous to one another, as the Delphi technique
was implemented by email between the panellists and the project managers, rather than
through face-to-face discussion. This was considered important as it gave equal weight to
each panellist’s judgments, avoiding problems that the power imbalances among the
panellists (for example, between students and faculty) might otherwise create. No
information was provided on what was integrated.
The Delphi questionnaires were distributed and responses received by email. Round one
was the open-ended question: ‘After reading the Environmental Policy, what recommendations
do you have to incorporate it into the activities and structure of Dalhousie University?’
A master list of 125 suggestions was developed from the responses. In round two, the
participants were asked to review the master list from round one and rate each item for
desirability and feasibility (separately) on a five-point Likert scale. The responses to
the second round were analysed statistically for measures of central tendency and
dispersion. The items were categorised as those that received consensus: a) for being
desirable and feasible; b) for being desirable but not feasible; or c) rated as either not
desirable or unsure. Each participant received a personalised questionnaire in round
three, listing that person’s ratings in round two, along with the group responses. They
were asked to reconsider their ratings and make any changes. In round three, the majority
of participants modified two to five of their round two ratings.
What was the outcome of the integration?
The results of the Delphi technique study were used by the university managers as a
key input to developing the Environmental Policy Implementation Plan. The features that
made the Dephi technique useful were identified as anonymity, encouraging exploratory
thought and developing innovative ideas, achieving consensus, serving as an educative tool
about environmental issues and being a tool for empowerment (Wright 2006).

2. Public health: estimating the incidence of Salmonella
poisoning

What was the context for the integration?
Despite food poisoning through food-borne Salmonella infection
being an important public health problem, in the United Kingdom in the mid 1990s, official
statistics were not able to provide an accurate estimate of the incidence of infections.
It was agreed that the official data significantly underestimated the true incidence, but
experts’ views differed about the level of underreporting.
What was the integration aiming to achieve and who was intended to
benefit?
The Delphi technique was used by Henson (1997:197) to: a) ‘reconcile differences in
expert opinion and provide more reliable estimates of the incidence of food-borne
Salmonella’; and b) to identify expert opinion about the
effectiveness of the available measures for control of the infection. This dialogue method
was chosen because it was, in the view of the person who implemented the study,

a recognised technique for reconciling differences in group
judgements where there is inherent uncertainty as to the actual state of the world. In
this case, the group consists of experts on food-borne Salmonella
in the UK. The aim is to generate data which may overcome acknowledged problems with
published statistics. (Henson 1997:196)

Who did the integration, how was it undertaken and what was
integrated?
The Delphi study was initiated by conducting a workshop in which seven experts in
food-borne Salmonella infection examined the issues to be covered in
the survey. They did so using the nominal group technique, discussed below. The workshop
identified the precise wording to be used in the Delphi study questions.
Some 62 experts (their areas of expertise were not specified) in food-borne
Salmonella infection, identified by workshop participants, were
then invited to be part of the Delphi study, and 42 of them agreed to do so. Five Delphi
rounds were conducted during a seven-month period, with three exploring the experts’
judgments of the incidence of infection and all of them exploring the effectiveness of
control measures. This was done by means of questionnaires, but further details were not
given.
The first question was: ‘What would you estimate to be the total number of persons ill
due to infection with non-typhoid Salmonella in the UK from all
sources (food and non-food), over the course of one year?’ The second asked what
proportion of infections participants thought was food-borne and the third invited them to
identify the proportion of cases by type of food. For each question, they were asked to
advise how they produced their estimates and any difficulties they encountered in doing
so. The results of the first and second rounds were fed back to participants, showing them
the median, minimum and maximum responses from the whole panel and inviting them to revise
their estimates of incidence.
In round one, participants were also asked to list the control strategies available
for reducing the incidence of food-borne Salmonella infection. In
round two, they were asked to refine the list and in round three the refined list was
presented along with the question, ‘Taking each control strategy in turn, consider how
effective it would be at reducing the total incidence of food-borne non-typhi
Salmonella in the UK?’ This question was repeated in the fourth and fifth
rounds, with the findings of the previous round fed back to participants.
What was the outcome of the integration?
An important outcome of the process was the narrowing of the range of estimates for
the incidence of infection, as participants reflected on the median and range of responses
to the incidence questions. Regarding the effectiveness of control measures, one approach
(food irradiation) was identified by the panel as being particularly effective.
Considerable disagreement remained, however, about which other measures were effective,
even after three rounds considering this question. The author concludes that this is not
really problematic as the Delphi study ‘provides a good summary measure of expert opinion
in an area which is characterised by great uncertainty’ and ‘the spread of responses
provides a good indication of the range within which we can expect to find the actual
state of the world’ (Henson 1997:203).

3. Security: developing a new medical school curriculum addressing bio-terrorism

What was the context for the integration?
Since the 11 September 2001 attacks on the New York World Trade Centre and the
Pentagon in the United States, responding to the medical sequelae of bio-terrorism and
biological warfare incidents is no longer considered solely the province of emergency
medicine specialists. Rather, it is seen as something that all healthcare providers need
to be prepared to handle.
What was the integration aiming to achieve and who was intended to
benefit?
Medical educators in the United States set out to develop new medical school
curriculum guidelines relating to bio-terrorism so as to equip the next generation of
medical graduates to be able to respond to this threat. They used an internet-based Delphi
survey to identify the educational objectives to be covered by the curriculum guidelines
(Coico et al. 2004).
The Delphi technique was chosen for this purpose because, in the views of those who
wished to develop the new curriculum guidelines, it ‘can provide a relatively rapid means
of gaining a consensus on complex issues’ (Coico et al. 2004:367).
What was being integrated?
This consensus came through the integration of the judgments of a group of experts in
microbiology and immunology, who were engaged in medical education in US universities.
Some 89 per cent of panellists had PhD degrees, 7 per cent were physicians and 77 per cent
were involved in medical curriculum development. Two-thirds rated their expertise
concerning bio-terrorism, biological warfare and bio-defence as ‘strong’ or ‘moderate’.
Who did the integration and how was it undertaken?
A total of 237 people were invited, by email, to join the Delphi panel and 64 (25 per
cent) participated in one or more rounds. The Delphi process comprised three
internet-based rounds using ‘a dynamic Web-based questionnaire’ (Coico et al. 2004:367).
The responses were captured from the web server onto spreadsheets. Before the first round,
participants provided demographic information including self-assessment of their expertise
in bio-terrorism.
Previous workshop discussions had produced a list of six content-related curriculum
categories for bio-terrorism teaching and learning: general issues, bio-defence, public
health, infection control, infectious diseases and ‘weaponizable toxins’ (Coico et al.
2004:368). These were put to the participants in the first Delphi round, and they were
asked to add knowledge, skills and attitude objectives to the list of educational
objectives. They were also asked for suggestions about any content areas that seemed
irrelevant to the project. In round two, the responses to round one were fed back to
participants and they were asked to assess, for three identified levels of medical
training, the relative importance of each objective. The results of round two were fed
back to the panel in round three in the form of percentage endorsed figures, and they were
asked to identify their top five curriculum objectives in each category. They were also
asked to rate the usefulness of nine different methods of teaching/learning and assessment
of bio-terrorism and bio-defence topics.
The products of round two were also passed to an independent expert committee to
obtain their views. This separate, independent committee had members who were experts from
other professions and disciplines concerned with the issues being addressed by the panel.
Its function was to receive the panel’s findings and consider their implications.
What was the outcome of the integration?
Although the authors of the paper reporting on this project stated that they would
have benefited from a higher participation rate, they felt that the Delphi technique
‘provided an opportunity to explore bioterrorism-related curriculum issues in depth’
(Coico et al. 2004:372). The outcome was the inclusion, in the US Medical Licensing
Examination, of approximately one-third of the educational objectives identified through
the Delphi study.

4. Technological innovation: developing professional association policies and
practices for shifting from paper to electronic communications

What was the context for the integration?
The Institute of Electrical and Electronics Engineers (IEEE) describes itself as the
world’s leading professional association for the advancement of technology, and the
largest, with more than 365 000 members in more than 150 nations
(<http://www.ieee.org/web/aboutus/home/index.html>). In the late 1990s, it
identified the need to establish policies and procedures governing its transition from
hard copy to electronic communication and dissemination of information within the
institute and beyond it. Indeed, in 1996, it adopted the slogan ‘IEEE: networking the
world’.
What was the integration aiming to achieve and who was intended to
benefit?
The institute used the Delphi method to assess the benefits of and obstacles to its
transition to electronic communications. It saw this method as a

technique that is considered appropriate when the research purpose
is to glean and synthesize expert opinion about complex issues and to identify
recommendations for addressing them. The technique is frequently used in exploratory
research and in efforts aimed at technological forecasting, including technological
trajectories and the impacts of technological change…Use of the method in this research
project has allowed the researchers an opportunity to pool a wide range of expert
opinion in order to arrive at a series of focused predictions that may guide the IEEE’s
approach during this significant transition period. (Herkert and Nielsen 1998:80)

Who did the integration, how was it undertaken and what was
integrated?
A pool of institute members—the exact number was not reported—was identified by the
project managers and invited to participate in the study. They came from five areas:
institute leadership and staff, institute technical activities representatives, institute
regional activities representatives, customers and ‘informed others’. Forty agreed to
participate in the study and 30 provided demographic information and responded to round
one. (It is not clear if the Delphi questionnaires were distributed and returned
electronically or in ‘pen-and-paper’ format.) In round one, participants were asked to
assess:

the potential contribution of electronic communication and
information dissemination in fulfilling the institute’s strategic planning goals and
objectives that did not rely explicitly on the use of electronic media

the impact of electronic communication and information
dissemination with respect to the five strategic planning goals and objectives that
relied explicitly on the use of electronic media.

They were also invited to provide open-ended commentary on the benefits of and
obstacles to the use of electronic media (Herkert and Nielsen 1998:82–4). A round one
example question was:

Products and Services Objective: Make all IEEE information
products and databases of value to members available in electronic form as quickly as
possible.

I agree; making
products and databases available in electronic

form as
quickly as possible is a valuable objective.

I disagree;
making products and databases available in electronic

form as
quickly as possible is not a valuable objective.

Discuss your
answer in the space provided below.

In round two, the panellists were given a synthesis of the obstacles to the
Institute’s increasing reliance on electronic communication derived from round one and
were asked to identify the 10 obstacles that each respondent considered most problematic.
In round three, they were provided with a list of the 11 major obstacles identified in
round two and asked what actions the Institute should take to benefit maximally from
electronic communications while avoiding its potential pitfalls. Content analysis was
conducted on the responses to round three to identify the path forward.
What was the outcome of the integration?
This application of the Delphi technique resulted in the IEEE identifying six key
factors affecting the adoption and use of electronic media:

characteristics of the IEEE as technology initiator;

characteristics of the potential individual adopter;

characteristics of the potential organizational adopter;

characteristics of the technology;

outcomes, and

characteristics of the contextual environment. (Herkert and Nielsen
1998:95–6)

This finding, combined with a content analysis of the panellists’ qualitative
responses, enabled the investigators to develop a range of recommendations for
consideration by the executives of the Institute to guide it in embracing electronic
communication methods.

Commentary

The Delphi technique is generally implemented by means of pen-and-paper, email or
web-based questionnaires, or by one-on-one interviewer–interviewee questionnaires. This
means that it does not entail face-to-face dialogue. Instead, a ‘conversation’ occurs by
means of responding to the questionnaires and sharing all the participants’ responses, one
with another. What are missing are the additional communication cues—verbal and
non-verbal—that occur in face-to-face dialogue. Here, the focus is on the contents of the
message, the real wording, rather than the other features that constitute human
communication. Nonetheless, we classify it as a dialogue method in that the iterations in
the process have features similar to two-way communication in the face-to-face situation.
All four case examples applied the Delphi technique reasonably closely to the ‘ideal
type’. The number of iterations used varied, with three of the examples utilising three
rounds and another (the public health example) five rounds—an unusually large number. This
also demonstrates how it can be implemented flexibly, depending on the topic, the
participants, resource considerations, and so on.
Most commonly, the method is used with a group of peers: experts with relatively equal
status, a more-or-less common knowledge base and a shared epistemology. This was the case
with the public health and security examples, all the participants in which were experts. In
contrast, the first example illustrated the participation of three stakeholder groups, none
of which was particularly expert on the topic. The fourth example demonstrates the method’s
use among a fairly diverse range of participants.
As a method of research integration, it is especially useful for complex problems about
which uncertainty exists and for which expert judgment is needed to deal with this
uncertainty. The problems are typically multifaceted and demand insights derived from
different types of knowledge, experience and information. This means that the problem being
addressed needs to be tightly defined and the questionnaires must deal explicitly with
boundary issues. It is a highly task-oriented process, seeking answers to a tightly defined
problem.
As the examples illustrate, the method is highly adaptable in terms of its contents. We
are aware of at least one example of the method being given a title that reflects the
contents being judged—namely, the ‘Ethical Delphi’ (Millar et al. 2006, 2007). This is not
so much a methodological variant as the application of the standard Delphi technique to a
particular content area—in this case, concerns about ethics and values.
This method, unlike those discussed above, relies very much on the people (the research
integrators) who manage the process to make the syntheses and judgments. They develop the
questions, score the responses and identify the conclusions, their validity and reliability
and their utility. This allows for strong focus on the task (in contrast to unfacilitated
face-to-face group processes where the focus can be readily diverted).

Origins and genealogy

The Delphi technique had its origins in the early 1950s’ Cold War between the United
States and the Soviet Union, when the RAND Corporation was commissioned by the US Department
of Defence ‘to apply expert opinion to the selection, from the point of view of a Soviet
strategic planner, of an optimal US industrial target system and to the estimation of the
number of A-bombs required to reduce the munitions output by a prescribed amount’ (Dalkey
and Helmer 1963:458). Its title refers to the Delphic Oracle, reflecting the fact that it
was originally used as a forecasting technique with respect to science and technology.
Since then, the Delphi technique has been used many thousands of times in diverse
sectors addressing a huge array of questions. Although the originators did not situate it
(in their 1963 paper) in any particular body of theory, subsequent scholars have attempted
to do so. A wide range of traditions in Western philosophy has been invoked in this context,
with one schema, presented by Mitroff and Turoff (1975), demonstrating that the method can
be understood through the Lockean Inquiring System (the basis of much empirical science),
the Leibnizian Inquiring System (the basis of much theoretical science), the Kantian
Inquiring System (which combines both of these approaches) and the Singerian-Churchman
Inquiring System.
Scholars have concluded that there is no single school of philosophy that best captures
the theory underlying the Delphi technique (Mitroff and Turoff 1975).

Further reading on the Delphi technique

Adler, M. and Ziglio, E. (eds) 1996, Gazing Into the Oracle:
The Delphi method and its application to social policy and public health,
Jessica Kingsley Publishers, London.

Linstone, H. A. and Turoff, M. (eds) 1975, The Delphi
Method: Techniques and applications, Addison-Wesley Publishing Company Advanced
Book Program, Reading, Mass.http://epress.anu.edu.au/dialogue_methods/mobile_devices/ch03s04.html ETHICAL DONATORS AND COMMUNITY MEMBERS
REQUIRED, TO FILL THIS SPACE WITH YOUR POLITICAL SLOGANS, ADVERTISING
OFFERS, WEBSITE DETAILS, CHARITY REQUESTS, LECTURE OPPORTUNITIES,
EDUCATIONAL WORKSHOPS, SPIRITUAL AND/OR HEALTH ENLIGHTENMENT COURSES. AS
AN IMPORTANT MEMBER OF THE GLOBAL INDEPENDENT MEDIA COMMUNITY, MIKIVERSE POLITICS HONOURABLY REQUESTS YOUR HELP TO KEEP YOUR NEWS, DIVERSE,AND FREE OF
CORPORATE, GOVERNMENT SPIN AND CONTROL. FOR MORE INFORMATION ON HOW YOU
MAY ASSIST, PLEASE CONTACT: themikiverse@gmail.com

AS AN IMPORTANT MEMBER OF THE GLOBALINDEPENDENT MEDIACOMMUNITY, MIKIVERSE POLITICSHONOURABLY REQUESTS YOUR HELP TO
KEEP YOUR NEWS, DIVERSE,AND FREE OF CORPORATE, GOVERNMENT SPIN AND CONTROL. FOR MORE INFO ON HOW YOU MAY ASSIST, PLEASE CONTACT: themikiverse@gmail.com