Abstract

Background The ability to consult effectively is key to the delivery of quality patient care. External peer review of consultations has been available to general medical practitioners (GPs) in the west of Scotland for several years. Pharmacists are expected to provide increasingly complex advice for patients. This study describes the development and first steps in the evaluation of a generic tool to be used to inform feedback about both GP and pharmacist consultations with patients. MethodThe toolwas developed by a small group of GPs and tested for validity using a content validity inventory. An item correlation pilot (ICP) was carried out. Three experienced reviewers reviewed and scored three consultations in each of six GP tapes, a further three reviewers reviewed and scored three consultations in each of four pharmacists’ tapes. The results underwent statistical analysis. ResultsPharmacists do not examine patients and so two similar instruments were developed, whereby two questions pertinent to patient examination were omitted from the pharmacist instrument. The content validity inventory demonstrated aspects of face validity and content validity of instruments. Statistical analysis of GP tapes suggested that the instrument could discriminate between GP consultations. Skills demonstrated by pharmacists were given lower scores, were less consistent than the GPs and the instrument did not discriminate between those performing at different levels. ConclusionThe results suggest that the instrument is useful only when applied to practitioners who have been taught how to consult in a patientcentred manner, and have led to the introduction of training in consultation skills for pharmacists throughout Scotland. Potential reliability in providing peer feedback for GP consultations is important in the context of the provision of consistent, meaningful evidence for GP appraisal and revalidation in the UK.

Keywords

Introduction

It has been
clearly established that the delivery of effective doctor–patient
communication not only en-hances patient satisfaction, but can also lead to
improved clinical outcomes.[1] By contrast, poor communication can
adversely affect clinical outcomes, patient satis-faction and levels of
litigation.[2] Communication skills and the ability to consult
effectively are key to the delivery of quality patient care. This has
been recog-nised by the General Medical Council (GMC)[3] and the Royal
College of General Practitioners (RCGP),[4,5] and is one the domains
requiring evidence in the General Practitioner Appraisal and Revalidation
Framework.[6] The introduction of appraisal in the UK and the debate
over what evidence will be necessary to satisfy the requirements of revalidation[7] have raised the ques-tion of how doctors can access education in this area and
provide objective and verifiable evidence of their ability to communicate
effectively with patients.[8]

Owing to their new contractual obligations, com-munity pharmacists in
the UK[9] are working in a rapidly evolving and changing clinical
environment and are now expected to provide increasingly complex advice to
patients in the primary care setting. To date, review of and training for
community pharmacists in communicating with patients about clinical issues has
been minimal.[10]

Since Balint[11] in the 1950s, many doctors and physiologists[12–15] have analysed the doctor–patient con-sultation. Perhaps the work carried out by
Pendelton et al[16] in the 1980s was most influential because he was the
first to describe a method of teaching the consul-tation using video. This has
influenced GPs in training for the last two decades, and consequentially a
large proportion of GPs working in the UK today have learned about patient-centred
consulting using this method. It has, however, been established that once
communication skills have been developed they need to be practised or they may
wither.[17]

One method of obtaining evidence about the qual-ity of communication
skills in the primary care setting is through external Peer Review. Peer Review
has been described as the critical evaluation of a specific aspect of a
practitioner’s performance by professional col-

leagues,
facilitated by a structured instrument whose utility reflects the purpose for
which it is intended.[18,19]

External Peer Review has been described as an essential stimulus to effective
performance and it has been suggested that it should become an integral part of
self-regulation.[20]

In the west of Scotland, established GPs have been able to voluntarily
submit videotaped consultations for external Peer Review as part of their
continuing personal development (CPD) for 10 years (Box 1). A consultation Peer Review group (CPRG) has been established. It has become clear to the
reviewers that consultations submitted for review by established GPs
differ from those submitted for assessment by GPs in training. The
consultations tend to be more realistic, often more complex, everyday general
practice which did not fit into the performance criteria-based assess-ment tool
used by Membership of Royal College of General Practitioners (MRCGP)
examination. The 18 active reviewers in a CPRG are trained in assessing the
quality of consultations and writing individual, de-scriptive feedback using a
tool based on a patient-centred model.[21] However, this instrument
has not previously been subject to adequate evaluation. This study describes
the first steps in the development and preliminary psychometric evaluation of a
generic tool to be used to inform feedback about both GP consul-tations and
clinical discussions between pharmacists and patients in the primary care
setting.

Method

Development
of tool

Four GP educationalists, with experience in
assessing GP consultations, used their professional and teaching expertise,
together with accumulated knowledge of patient-centred consulting,[14–16] to analyse the content of patient–professional interactions. By round the table
discussion and professional consensus, relevant questions were agreed and
collated. The aim was to be as inclusive as possible, as it was acknowledged
that poor questions could be detected and removed, but there was no recompense
for any items not initially included. These questions resulted in development
of items for proposed feedback instrument.

Content
validity exercise

To demonstrate
that the important areas of the con-sultation were being considered in the
proposed instru-ment a content validity exercise[22] (CVE) was
performed. The proposed instrument was sent, by post, to a convenience sample
of 10 GPs and pharmacists. All recipients were well informed about consulting
techniques and work
in primary care in Scotland. They consisted of six GPs from the existing group
of Peer Reviewers, two GPs with a special interest in consul-tation skills,
together with two primary care pharma-cists with a background in CPD. All 10
professionals approached to take part in CVE agreed to provide responses. Each
respondent rated the perceived rele-vance of each item to a patient-centred
consultation using a Content Validity Index (see Appendix A online at www.radcliffe-oxford.com/journals/J10_Quality_ in_Primary_Care/Supplementary_Papers.htm>) and returned
their response by post. This rating exercise underwent an iterative process
until 80% agreement for each item of the proposed instrument was demon-strated.
The identified items formed the content of the feedback instrument to be tested
further. A seven-point Likert-type[23] rating scale was agreed for
each item to enhance reliability in discrimination.

Item
correlation pilot

To assess the
contribution of each item in the proposed instrument and to establish
consistency, an item corre-lation pilot (ICP) was performed. Multicentre
Ethical Approval was obtained. A convenience sample of six GPs, who had
voluntarily submitted consultations for Peer Review under the existing CPD
mechanism, consented to take part in this project.

The authors ran workshops at several pharmacy educational meetings,
with the aim to identify six volunteer pharmacists who were prepared to make
and submit videotapes of their consultations with patients. GPs and pharmacists
obtained informed consent from all participating patients both before and
immediately after consultation.

All established consultation Peer Reviewers in the CPRG, who were not
involved with the development of the instrument, were invited to take part in
reviewing consultations for this project. Six volun-teered and were selected
for the task.

At a central venue, three reviewers assessed and scored the same three
consultations on each of the six GP tapes, and three other reviewers assessed
and scored three identical consultations on each of the pharmacist tapes.

The following statistical
analysis was undertaken:

flassessors – Limits of agreement were
calculated to give a range within 95% of the differences between which
the assessors’ scores would lie, both for those assessing GP consultations and
those assessing phar-macist consultations.

• GP consultation skills – The score each
GP was given for each item by each of the three assessors was plotted for each
consultation.

Results

It was
acknowledged at the time of initial development of the instrument that
pharmacists do not examine patients. Although an identical generic tool could
not be employed to assess the patient centredness of both GP and pharmacist
consultations, by simply removing the two items relating to patient examination
in the pharmacy tool, two similar assessment tools were able to be developed
and tested concurrently (see Appendices B(GP) and B(Ph) online at www.radcliffe- oxford.com/journals/J10_Quality_in_Primary_Care/ Supplementary_Papers.htm>).

Content
validity exercise

• The proposed GP instrument consisted of
24 items each rated on a seven-point Likert scale, 23 ques-tions relating to
the patient-centred consultation and 1 giving an overall score.

• The proposed pharmacist instrument
consisted of 22 of these 24 items each rated on a seven-point Likert scale, 21
questions relating to a patient-centred consultation and 1 overall score.

flan agreement of 80% was demonstrated
for in-clusion of all 24 items using the proposed tool after one round of the
CVE. There were some minor adjustments made to the wording of individual items
as a result of this exercise.

Item
correlation pilot

GP consultations

Three
consultations from each of six GP tapes were viewed independently by three
reviewers, i.e. each re-viewer watched and assessed 18 identical consultations.

STATISTICAL
ANALYSIS OF GP TAPES

• Assessors:

– Inter-rater consistency was
measured by assessing the agreement between rater’s scores for each item
included in the instrument. The scores of assessor 1 and assessor 2 were within
1 point (on the Likert scale) of each other for 83% of the items. Assessors 1
and 3 agreed within 1 point for 76% of the items and assessors 2 and 3 agreed
within 1 point for 63% of the items. No assessor differed on average from
the other two

by more
than 0.7; there was good agreement between assessors.

• GP consultation skills (Table 1):

– Reviewer’s qualitative
judgements regarding those GPs who performed exceptionally well (GP5) and those
who performed less well (GP6) were backed up by a statistical difference
in their mean scores:

Only four
pharmacists were able to submit tapes in the allotted period, three
consultations from each of four pharmacist tapes were viewed independently by
three GP reviewers, i.e. each GP reviewer watched and analysed 12 identical
consultations.

STATISTICAL
ANALYSIS OF PHARMACIST TAPES

• Assessors:

assessor 1 and assessor 2 were within 1
point (on Likert scale) of each other for 77% of the items. Assessors 1 and 3
agreed within 1 point for 67% of the items and assessors 2 and 3 agreed within
1 point for 73% of the items.

• Pharmacist consultation skills (Table 2):

– A simple mean of the scores
for each pharmacist was used to summarise their overall perform-ance. The
individual performances were fairly similar, with no pharmacist appearing to
per-form exceptionally well or poorly. The numerical values are lower than
those noted in equivalent GP assessment.

Discussion

Summary
of main findings

This study has
outlined the groundwork and first steps in developing and testing an instrument
to be used to deliver effective, consistent written feedback for established
GPs and community pharmacists working
in the general practice setting. Previous studies have
focused on assessing GP consultation competence– Inter-rater consistency was
measured by assess- tablished GPs and community pharmacists working ing the
agreement between rater’s scores for each in the general practice setting. Previous
studies have item included in the instrument. The scores of focused on
assessing GP consultation competence using
standardised patients,[24,25] or within the training environment.[26] Although the initial aim was to de-velop a generic tool, two similar assessment
tools were developed and tested, one to rate the patient-centred consulting
skills of GPs and the other to rate the patient-centred consulting skills of
pharmacists.

The aim of consultation Peer Review is to provide effective
written feedback, within the limitations of this preliminary, small study the
content of both tools was demonstrated to show certain aspects of both face
validity and content validity. To underpin consistency in this feedback, it is
necessary to establish reliability of the instrument. This study has begun to
explore aspects of internal consistency and inter-rater reliability. The
results, although limited, indicate that when the tool was applied to the GP
tapes, the reviewers demon-strated concordance. It was also indicated that a
simple mean of the recorded scores could consistently rep-resent the GPs’
overall performance in patient-centred consulting and that this was consistent
over three consultations. When the simple mean was compared with the informed
judgement of the reviewers it was possible to discriminate between the
differing levels of skill demonstrated by the GPs. Agreement was less
evident in the pharmacist tapes, evaluation of per-formance was less consistent
and generally the scores were lower and more widespread.

The initial method proposed the inclusion of a reliability pilot, to be
undertaken once the ICP was completed and analysed, to assess the ability of
trained reviewers in using the instrument in a consistent manner. However,
analysis of the ICP demonstrated that the level of agreement differed
when comparing GP tapes and pharmacist tapes; being significantly greater when
used for the GP tapes. This evidence demonstrated that the tool did not
discriminate at consistent levels with each professional group, and therefore
the reliability pilot was not pursued.

Sensitivity analysis suggested that several of the items used in each
tool could be omitted with no effect on overall sensitivity of the
instrument. How-ever, to enhance the writing of consistent, effective,
constructive feedback the authors feel it is important for the reviewers to
consider all areas of the consul-tation.

Strengths and limitations of study

NHS Education
for Scotland (NES) aims to encourage health professionals to develop
interprofessional learn-ing.[27] In this study, GPs and pharmacists
worked together achieving strong links and understanding between the
professions.

The study took place in Scotland, pharmacists and GPs were recruited
from the west of Scotland, re-viewers and those involved in CVE were from all
areas of Scotland. The
consultation is a core element of primary care in the UK and these results are
therefore relevant to all areas of the UK.

The instruments were initially developed and tested only by GPs. The
absence of pharmacist involvement in the initial question development may have
biased the results of the CVE. The authors were involved in the training of
postgraduate pharmacists in com-munication skills at the time of the study and
were aware that relevant expertise was not available within the pharmacy
profession in Scotland. The ‘experts’ employed were highly experienced in
consultation skills training, but the number of individuals with
sufficient knowledge and expertise to be deemed appropriately skilled in
this area is limited.

Both GPs and pharmacists submitting their consul-tations for this
project were volunteers. All assessors were GPs, at the time of the study only
GPs had been trained in consultation Peer Review; this meant that the
phar-macist consultations were assessed from a GP per-spective. Assessors were
all members of the established CPRG, and as such were both confident and
experi-enced in reviewing consultations. Non-trained indi-viduals would
undoubtedly have been less consistent.

Available resources allowed only small numbers of participants to be
included in this pilot. Only four pharmacists were able to submit consultations
for review within the allotted time. Larger numbers of consultations or larger
numbers of assessors may have influenced the results.

Each assessor was asked to consider 22 or 24 individual questions about
each consultation, a task, which requires expertise and prolonged
concentration, sufficient resources were identified to ensure pro-tected
time and all assessments were undertaken in a venue with the required
facilities.

The pharmacists

Until now, few
pharmacists have received structured training or assessment in consultation
skills,[28] and this topic has only recently been tentatively
introduced into their undergraduate curriculum. By contrast, UK medical
students have been taught communication skills as part of their core curriculum
since the mid-1990s.[29] GP registrars have been taught consultation
skills since the early 1990s, and these skills have been assessed summatively
since 1996.[30]

The results suggested that the
tools employed demonstrated certain aspects of reliability when ap-plied to
participants who had received teaching in patient-centred consulting skills.
Reliability was more limited for those participants who had not received
teaching in this area. These outcomes have led to the introduction of training
in patient-centred consulting and the development of a consultation Peer Review process for
pharmacists throughout Scotland led by GPs and funded by NES.

The GP
context

Communication
with patients is one of the five core categories, to be undertaken on a
five-year cycle, in GP appraisal in Scotland.[31] However, it has been
demonstrated that fewer than 10% of those appraised and submitted evidence in
this area felt there had been any impact on their consulting as a result.[32] This requirement can be addressed in a variety of ways including the use of
standardised survey, e.g. General Practice Assessment Questionnaire (GPAQ).
Surveys such as GPAQ are a required annual element of the Quality and Outcomes
Framework. Although their value has been questioned, it is not surprising that
most GPs chose this method, this instrument, how-ever, has not been formally
tested for reliability.[33] The authors suggest that participation in
an effective Peer Review process may have greater educational and
developmental impact. Peer Review of consultations may also offer a more
feasible method of demonstrat-ing consultation skills for sessional doctors or
those in small practices, than the proposed reliance on multi-source feedback.[34]

The GMC Good Medical Practice Framework for Appraisal and Assessment
(Revalidation: the way ahead) for all doctors proposed that ‘validated tools
for feedback about doctors’ practice’ could provide the evidence required for
appraisal in all four domains (Box 2).[31] This is an example of
initial development of such a tool.

of performance. It is essential to also
consider accept-ability, feasibility and educational impact[35] both
for participants and reviewers. Further work is under way to evaluate this.

If this model was to be adopted
by community pharmacists, more work, including retesting would be required,
using consultations from pharmacists who have gained experience in
patient-centred consulting.

When providing feedback to peers, the largest element of variability,
in common with other instru-ments, derives from the difference between
individual assessors.[36,37] Significant additional work is required
to ensure acceptable reliability of the instrument, but in common with other
assessment instruments[38] ongoing calibration of assessors is
essential to optimise inter-rater reliability.

Previous work has suggested that because the con-sultation is the
cornerstone of medical practice, a central focus of revalidation should be the
assessment of consultation competence,[39] and discussion of the new
regulatory processes suggests that all doctors may be asked to provide evidence
of the effectiveness of their consultation skills for the purposes of
revalid-ation.[7] The results suggest that with further develop-ment
this tool could offer a method of objectively and reliably assessing a
GP’s performance in the consulting room. It would, however, be imperative to
ensure that adequate resources were available to provide training in this area,
to enable consistent feedback and that effective support was available
for those doctors whose performance gave cause for concern.

The
future

The key purpose
of consultation Peer Review is to facilitate educational feedback to GPs and
pharmacists on their demonstrated consulting skills. The results of this pilot
study highlight the validity and potential reliability of this tool in
providing consistent Peer Review feedback for GP consultations.

Considerable further work is required to confirm the overall utility of
the instrument. A simple mean of scores for GPs summarised their overall
performance, and was able to discriminate between different levels

The authors
would like to thank Dr Douglas Murphy and Dr Colin Hodgson for their help in
developing the content of the instrument, the CPRG for their reviews and all
participating GPs, pharmacists and their patients for the consultations
provided for review. Grateful thanks to Dr Lillian Murray, University of
Glasgow for statistical advice.

Acknowledgements

The authors would like to thank Dr Douglas Murphy
and Dr Colin Hodgson for their help in developing the
content of the instrument, the CPRG for their reviews
and all participating GPs, pharmacists and their patients
for the consultations provided for review. Grateful
thanks to Dr Lillian Murray, University of Glasgowfor
statistical advice.

Likert R. A technique for the measurement of attitudes. Archives of Psychology 1932;22(140).

Fraser RC, McKinley RK and Mulholland H. Consul-tation competence in general practice: testing the re-liability of the Leicester assessment package. > British > Journal of General Practice 1994;44:293–6.

Fraser RC, McKinley RK and Mulholland H. Consul-tation competence in general practice: establishing the face validity of prioritized criteria in the Leicester as-sessment package. > British Journal of General Practice 1994;44:109–13.

Campbell LM, Howie JGR and Murray TSM. Summative assessment: a pilot project in the west of Scotland. > British Journal of General Practice 1993; 43:430–4.

NHS Education for Scotland, 2005. > www.nes.scot. nhs.uk/disciplines

Waterfield J, Aspinall V and Hall S. The use of digital video media in teaching of communication skills. The Pharmaceutical Journal 2009;282:135–6.

General Medical Council. Tomorrow’s Doctors. London: GMC, 1993.

Campbell LM and Murray TS. Summative assessment of vocational trainees: results of a 3 year study. > British > Journal of General Practice 1996;46:411–14.

Contributors

RM led the study conception and design, and
orchestrated the research application, subject partici-pation, the consultation
review process and the data collection. AP coordinated recruitment of
pharma-cists. NC contributed considerably to both writing of the paper and to
revision of the manuscript.

Funding

This study was funded by NHS Education for
Scotland.

Ethical Approval

This study was approved by Scotland A Research Ethics
Committee, Edinburgh.