In this paper we examine candidate performance in the three components of the MRCP(UK) for variation in performance in relation to the institution of graduation. The main analysis is a multilevel model assessing whether graduates from different UK medical schools performed differently in the three components of MRCP(UK) from 2003 to 2005, with differences between medical schools then being assessed in relation to 'compositionalvariables' describing the schools. An additional analysis looks at a much longer time series of data for performance on the Part 1 examination only from 1989 to 2002.

Additional analysis

The data for the additional analysis were for Part 1 only from 1989/1 to 2003/1, as well as Part 1 data from 2003/2 to 2005/3.

Candidates were only included who had graduated from one of the 19 UK universities then awarding medical degrees. The formats of the Part 1, Part 2 and PACES stages of the examination were stable between 2003/2 and 2005/3. The Part 1 examination comprised two separate 3-hourpapers each of 100 test items in a one-answer-from-five (best-of-five) format. The written examination of Part2 comprised two separate 3-hour papers each of 100 questions in a one-answer-from-five (best-of-five) format until the last 2003 diet when it increased to three 3-hour papers each of 90 questions. The PACES examination comprised a five-station, structured clinical examination lasting 2 hours, incorporating 10 separate clinical encounters each of which was directlyobserved and assessed by two different and experiencedclinician examiners, with each candidate being assessed by 10 examiners in total. There were three diets of Part 1, Part 2 and PACES each year. From 1989/1 to 2002/1 the Part 1 examination consisted of a single paper containing 300 multiple true-false items. From 2002/2 to 2003/1 the Part 1 exam consisted of a similar multiple true-false paper and a separate best-of-five exam with 100 questions.

Compositional variables

As the MRCP(UK) database does not contain data at candidate level on a number of measures (for instance, pre-admission qualifications, such as A-levels or Highers), data from several different sources were aggregated to provide 'compositional variables'. About 90% of candidates in the main analysis qualified between 1999 and 2003 (Part 1: 96.4% of 7763; Part 2: 93.1% of 4470; PACES 89.6% of 4147), hence most would have entered medical school between 1994 and 1998, and so where possible compositional data were found for cohorts as close as possible to that entry period. We acknowledge that a proportion of students would have taken intercalated degrees or, for other reasons such as exam failure, would have qualified perhaps 6 or even 7 years after entry to a medical school. However, the MRCP(UK) database only contains information on date of qualification.

Pre-admission qualifications

As mentioned above, results of A-levels and Highers for medical school entrants were not available. However, data on the pre-admission qualifications of medical school applicantsreceivingoffers were obtained from the UCASapplicant cohorts of 1996 and 1997, which are in the publicdomain, and have been extensivelyanalysed elsewhere [12]. Valid total pointscores for A-levels and Highers (excluding general studies) were converted to z-scores for the entirepopulation of applicants. Most applicants had only one of the two scores, which were used in the analysis, while for the minority of applicants with scores on A-levels and Highers, the higher of the two values was used as a measure of pre-admission qualifications. The UCAS database only included information on applicants receiving offers from particular medical schools, and aggregated means for individual medical schools were therefore calculated for all candidates receiving an offer at a particular school. It should be emphasized that since not all applicants receiving offers will subsequentlyenter a particular medical school, the correlation between average grades of applicants receiving offers and the average grades of entrants to a medical school will be less than perfect. Nevertheless the correlation is likely to be high.

Career interest in hospital medicine

Students in the 1991 cohort study used a five-point scale to indicate their interest in 28 specialities, both at application to medical school and in the final year [16]. Mean scores, aggregated by medical school, were based on 2813 and 1472 respondents, with a median per school of 92 and 62, respectively.

Proportion of graduates taking MRCP(UK) Part 1

Schools differ in the proportion of their graduates taking MRCP(UK). The number of graduates from each school known to have taken MRCP(UK) Part 1 at their first attempt during the eight diets from 2003 and 2005 was expressed as the percentage of the number of students from each school who registeredprovisionally with the GMC from 2001 to 2003 (data provided by CHMS; see Additional file1). In Cambridge, Oxford and Edinburgh, 40%, 40% and 38% of graduates, respectively, took MRCP(UK), compared with 27%, 24% and 23% of graduates of Liverpool, Leicester and Birmingham, respectively.

Performance at MRCGP

MRCGP (Membership of the Royal College of General Practitioners) is the principal postgraduate assessment for doctors in the UK wishing to become general practitioners. The percentage of graduates from each medical school who passed MRCGP at the first attempt between 1988 and 1991 is available from the study of Wakeford et al. [17]. Recent data for the period 2003–2006 are also available in the recent paper of Wakeford et al. [18].

Results

Main analysis

In the main analysis, results were available for 5827 candidates, 4040 of whom took Part 1, 3467 took Part 2 and 2888 took PACES for the first time, with 1248 taking all three parts within the time period, 2072 taking two parts and 2507 taking a single part.

Multilevel modelling

Effect of background variables

The multilevel model examined scores in the three parts of the examination in a single model, simultaneouslyestimating both the variances of the scores and their covariances (using information from those candidates taking two or more parts). The main interest was in differences between medical schools. However, background variables were also significant. Males performed better at Part 1 and Part 2 (Part 1: t = 2.863, p = 0.0042; Part 2: t = 2.281, p = 0.0010) and females performed better at PACES (t = 5.777, p = 7.6 × 10-9); white candidates performed better at all three parts (Part1: t = 2.789, p = 0.0054; Part 2: t = 2.561, p = 0.010; PACES: t = 4.333, p = 0.000014). Candidates who had been qualified for a longer time performed significantly worse at Part 1 (t = 4.393, p = 0.000011), Part 2 (t = 32.1, p < 10-12) and PACES (t = 4.471, p = 0.000007); see Figures S1–3 in Additional file 1. Candidates who had a longer delay between Part 1 and Part 2 performed better at Part 2 (t = 8.175, p = 2 × 10-16), although there was no significant effect of the delay between Part 2 and PACES on PACES performance (t = 1.1, p = 0.271).

Correlations between examination parts

There were highly significant correlations at the candidate level for performance on first attempts at different parts of the examination (Part 1 with Part 2, r = 0.600, n = 2492, p < 0.001; Part 1 with PACES, r = 0.247, n = 1250, p < 0.001; Part 2 with PACES, r = 0.260, n = 2074, p < 0.001; see Figures S4–6 in Additional file 1) indicating a reasonably high degree of stability in performance within individuals.

Medical school effects

Multilevel modelling found a highly significant overall effect of medical schools (χ2 = 300.57, six degrees of freedom, p < 0.001), with significant variance between schools for Part 1 (variance = 8.345, standard error (SE) = 2.85, t = 2.928, p = 0.0034), Part 2 (variance = 2.557, SE = 0.916, t = 2.791, p = 0.0053) and PACES (variance = 0.787, SE = 0.327, t = 2.401, p = 0.016). (The full fitted multilevel model is shown in Figures S7 and S8 in Additional file 1). Medical school variance is greatest for Part 1 and least for PACES, but those differences also reflect differences in total variance (82.6 for Part 1, 40.4 for Part 2 and 28.4 for PACES). The coefficient of variation, expressed as medical school SD as a percentage of total SD, is 31.7% for Part 1, 25.1% for Part 2 and 16.6% for PACES.

Figure1shows the residuals for each part, rankordered in each case by the effect found in Part 1 of the examination. In Part 1 of the examination and following correction for multiple testing, the graduates of Oxford, Cambridge and Newcastle-upon-Tyne performed significantly better than average, and the graduates of Liverpool, Dundee, Belfast and Aberdeen performed significantly worse than average.

Figure 1

Medical school effects for the Part 1, Part 2 and PACES exams of MRCP(UK). For all three parts of the examination, medical schools are sorted by the size of the effect at the Part 1 examination. Error bars indicate ± 1 SD. Note that absolute values are different for the three examinations (see the text).

At the medical school level, performance at Part 2 correlated significantly with performance at Part 1 (r = 0.981, p = 0.004), with the same schools as for Part 1 showing significant differences from the mean.

In the PACES examination, the correlation with performance at Part 1 and Part 2 was a littlelower than that found between Part 1 and Part 2, but was also highly significant (Part 1 with PACES: r = 0.849, p = 0.0114; Part 2 with PACES: r = 0.897, p = 0.0096). Four schools performed significantly differently from average, three of which were also significant at Part 1 and Part 2 (Oxford above average, and Dundee and Liverpool below average) and in addition London also performed significantly worse than average, although London graduates had been almostprecisely at the average for Parts 1 and 2. (Scattergrams of the relationship between medical school effects at Part 1, Part 2 and PACES can be found in Figure S9 in Additional file 1).

Analysis of compositional variables

In this section we analyse data at the level of the 19 medical schools, and wheneverphrases such as 'higher pre-admission qualifications' are used it must be emphasized that this refers to 'medical schools whose candidates have higher pre-admission qualifications' and does not mean 'individual candidates with higher pre-admission qualifications'. Correlations and structural models at the individual and school level may be similar but they need not be [20], and the analyses described here are specifically at the school level of analysis.

Table1 shows correlations between a school's average performance in the three parts of the exam and the compositional variables describing the school (for more details see Additional file 1). The highest correlations with MRCP(UK) performance are with pre-admission qualifications (see Figure 2), the correlation between aggregated mean pre-admission qualification and aggregated mean performance at Part 1 being highly significant (r = 0.779, n = 19, p < 0.001) and remaining significant when Oxford and Cambridge are omitted from the analysis (r = 0.566, n = 17, p = 0.018).

Figure 2

Average pre-admission qualifications of applicants receiving offers at UK medical schools. Values are z-scores of A-levels and Highers and are standardized across all applicants (and hence those receiving offers tend to have above average values). Error bars indicate ± 1 SE and because sample sizes are large (typically of the order of 500 and over 5000 in the case of London), error terms are small (see the text).

Table 1

Part 1

Part 2

PACES

Mean pre-admission qualifications at A-levels or Highers

0.779p = 0.000085

0.773p = 0.000011

0.704p = 0.00076

Interest in career in hospital medicine at application

0.205p = 0.401

0.196p = 0.421

0.223p = 0.358

Interest in career in hospital medicine in final year

0.510p = 0.026

0.522p = 0.022

0.500p = 0.029

Interest of teaching in general medicine

0.588p = 0.0081

0.568p = 0.011

0.483p = 0.036

Difficulty of teaching in general medicine

0.128p = 0.603

0.143p = 0.559

0.153p = 0.532

Usefulness of teaching in general medicine

0.223p = 0.358

0.234p = 0.334

0.219p = 0.369

More time needed for teaching of general medicine

0.023p = 0.926

0.009p = 0.972

-0.049p = 0.842

Percentage of graduates taking MRCP(UK)

0.613p = 0.005

0.575p = 0.010

0.478p = 0.038

Pass rate at MRCGP, 1988–1991

0.601p = 0.0065

0.611p = 0.0054

0.532p = 0.019

Pass rate at MRCGP, 2003–2006

0.690p = 0.0011

0.726p = 0.0004

0.792p = 0.00005

Correlations of medical school performance and compositional variables

Although medical schools with a higher proportion of graduates taking MRCP(UK) tended to have higher pre-admission qualifications (r = 0.833, p = 0.001, n = 19), there was a weaker correlation between a medical school's performance at MRCP(UK) and the proportion of its graduates taking the exam (r = 0.613, p = 0.005, n = 19). The proportion of graduates taking MRCP(UK) did not predictoutcome after pre-admission qualifications were taken into account (β = -0.175, p = 0.559), whereas pre-admission qualifications did predict outcome after taking into account the proportion of graduates taking MRCP(UK) (β = 0.928, p = 0.006). There is therefore no independent effect of the proportion of a school's graduates taking MRCP(UK).

The relationship between all of the variables and Part 1 performance was examined using multiple regression, and only pre-admission qualifications predicted performance at MRCP(UK). The structural equation model in Figure 3 shows that the only variable with a direct or indirect effect on Part 1 or PACES performance is higher pre-admission qualifications, which had separate effects with teaching being rated as more interesting, a greater career interest in medicine in the final year and a higher proportion of graduates taking MRCP(UK).

Figure 3

Structural equation model for the causal relationship between the variables at the medical school level. Path strengths are shown as ∃ (standardized) path coefficients and significance levels based on a t-statistic with 17 degrees of freedom. The width of paths is proportional to the path coefficient. The saturated model allowed all variables to the left of a variable to have a causal influence on that variable and non-significant paths were removed until paths remaining were significant with p < 0.05. Paths not shown as causal arrows did not reach significance with p < 0.05.

Of particular theoretical interest (see the discussion) is that the performance of a medical school's graduates at MRCP(UK) correlated highly with performance in the MRCGP when taken in 2003–2005 and a little less so with the performance in 1988–1991 (see Table 1).

Performance in relation to the Guardian assessments

Table 2 shows correlations between the variables reported in the two compilations of data by the Guardian, and outcome at Part 1, Part 2 and PACES. The highest correlations, for both sets of data, are with the entry scores, which are based on university admission criteria. Using a forward entry multiple regression, in which the entry score based on the 2003–2004 data was entered first, no other variables apart from university admission criteria were significant predictors of Part 1, Part 2 or PACES performance.

Table 2

Years data are mainly based on

Guardian scores

Part 1

Part 2

PACES

2005–2006

Overall score

0.578p = 0.010

0.558p = 0.013

0.398p = 0.092

Teaching score (based on National Student Survey) (Note:N = 14)

0.218p = 0.454

0.242p = 0.405

0.262p = 0.366

Feedback score (based on National Student Survey) (Note: N = 14)

-0.101p = 0.731

-0.048p = 0.871

0.088p = 0.765

Spending per student

0.414p = 0.078

0.363p = 0.127

0.131p = 0.594

Staff/student ratio

0.001p = 0.998

-0.108p = 0.941

-0.077p = 0.755

Entry score (based on UCAS tariff scores)

0.603p = 0.006

0.612p = 0.005

0.572p = 0.011

2003–2004

Overall score

0.540p = 0.017

0.486p = 0.035

0.308p = 0.200

Staff score

0.118p = 0.630

0.081p = 0.741

0.035p = 0.888

Spending per student

0.499p = 0.030

0.445p = 0.056

0.239p = 0.325

Staff/student ratio

0.223p = 0.358

0.191p = 0.434

0.128p = 0.601

Entry score (based on UCAS tariff scores)

0.561p = 0.013

0.590p = 0.008

0.581p = 0.009

Correlations of medical school performance and Guardian scores.

Additional analysis

Data were available for a total of 22453 graduates of UK medical schools taking the Part 1 exam for the first time in the 51 diets from 1989/1 to 2005/3. Figure 4 shows performance of candidates from different schools taking Part 1 in 1989–1992, 1993–1995, 1996–1998 and 1999–2001, with schools plotted in the order found in the main analysis for 2003–2005. Performance in 2003–2005 correlated 0.739 with performance in 1989–1992, 0.816 with performance in 1993–1995, 0.848 with performance in 1996–1998 and 0.884 with performance in 1999–2001 (n = 19, p < 0.001 for all correlations). Excluding Oxford and Cambridge, the correlations were 0.493, 0.618, 0.654 and 0.723, respectively (n = 17, p = 0.045, 0.008, 0.004 and 0.001). Detailed, year-by-year graphs of trends within individual schools can be found in Figures S10 and S11a-11e in Additional file 1.

Figure 4

Differences between medical schools for candidates taking Part 1 MRCP(UK) from 1989 to 2001. Medical schools are sorted by Part 1 performance in 2003–2005. Error bars indicate ± 1 SE.

redgrey

Discussion

Our analysis shows that candidates who have trained at different UK medical schools perform differently in the MRCP(UK) examination. In 2003–2005, 91%, 76% and 67% of students from Oxford, Cambridge and Newcastle passed Part 1 at their first attempt, compared with 32%, 38%, 37% and 41% of Liverpool, Dundee, Belfast and Aberdeen graduates, so that, for instance, twice as many Newcastle graduates pass the exam first time compared with Liverpool graduates (oddsratio = 4.3×).

At the medical school level, performance at Part 1 correlates almost perfectly with performance at Part 2 (and both are multiple-choice examinations), while performance at PACES, which is a clinical examination, still correlates highly with Parts 1 and 2, although there are some smallchanges in rank order, the most notable being that London graduates perform worse than average at PACES but not at Part 1 and Part 2.

School-leaving examinations are known at the individual level to predict performance in undergraduate medical examinations and in postgraduate careers [23,24]. Although pre-admission academic qualifications correlate significantly with MRCP(UK) Part 1 performance at the medical school level (r = 0.779), that correlation is substantially less than the correlation found between Part 1 and Part 2 of the examination (r = 0.992). Pre-admission qualifications therefore account for about 62% of the accountable variance, leaving about 38% of the school-level variance dependent on other, unknown, factors. It should be emphasized that because sex and ethnic origin have been entered into the multilevel model at an individual level, there can be no differences at medical school level attributable to ethnicity or sex.

There are at least three broadtypes of explanation for the differences we have found: differences in those entering the schools (selection effects); differences in education or training at the school (training effects); or differences owing to students from different schools preferring different postgraduate careers (career preference effects).

Selection effects would predict that better qualified students enter schools such as Oxford, Cambridge and Newcastle-upon-Tyne (and Oxford and Cambridge, in particular, have traditionallydemandedvery high A-levels), so that the better-qualified entrants to those schools would also be likely to perform better in postgraduate examinations. At the individual level it is known that A-level results correlate with performance in MRCP(UK) Part 1 [24] and there are also clear differences in the average pre-admission qualifications of applicants receiving offers at different medical schools (see Figure 2). Our analysis of compositional variables leaves little doubt that one-half or more of the variance between schools can be explained by differences in intake, and that is supported by the correlations found with the data reported in the Guardian tables, which are compiled from a range of official statistics (Table 2). However, even at Part 1 the correlation leaves at least one-third of the variance unexplained. In particular, MRCP(UK) performance is about one SD higher than predicted from pre-admission qualifications alone for Leicester, Oxford, Birmingham, Newcastle-upon-Tyne and London, and about one SD lower than expected for Southampton, Dundee, Aberdeen, Liverpool and Belfast. Neither can differences in pre-admission qualifications explain the relative underperformance of London graduates at PACES, compared with Part 1 and Part 2. Pre-admission qualifications are a part of the story, but are not the entire explanation of medical school differences and the remaining variance is most likely to be relatedeither to other differences in the intake of schools or to differences in the education provided by those schools.

Career preference effects would occur if the differential performance of graduates on MRCP(UK) reflects a form of self-selection into different specialities (and Parkhouse reported, for instance, that amongst those qualifying between 1974 and 1983 that hospital medicine was particularly popular for Oxford, London and Wales graduates, and particularly unpopular for Aberdeen, Dundee and Leicester graduates [25]). If popularity also equated to status and kudos, then it might be that the most academicallygifted students at one school might prefer to go into one particular speciality, whereas at another school they might prefer a different speciality. Candidates would then perform better if they came from schools where a higher proportion of graduates took the MRCP(UK). However, our data show that not to be the case, as the correlation of performance and the proportion taking the exam was non-significant after pre-admission qualifications are taken into account.

Career preference effects also predict that if training at all schools is on aggregateequivalent, then schools performing better at one particular postgraduate examination, because their better students prefer to take it, should also perform less well at other examinations which are taken by their less gifted graduates. Overall there would then be a negative correlation in the ordering of schools across anypair of postgraduate examinations. In a study of performance at MRCGP in the early 1990s [17], graduates of Oxford, Cambridge and Newcastle-upon-Tyne ranked1st, 5th and 7th in performance, compared with Belfast, Aberdeen, Dundee and Liverpool graduates who ranked 16th, 23rd, 24th and 26th out of the 27 UK medical schools, with an overall positive correlation of effect sizes of r = 0.480 (p = 0.038, n = 19). More recent data for the MRCGP from 2003–2006 show a similar and somewhatstrongertrend (see Table 1). Such positive correlations, if confirmed by other examinations, would make the career selection explanation unlikely.

Institutions can differ in the amount of 'value' that they add, an effect well known in secondary education [26]. Training effects would predict that teaching and training in general medicine at some schools is a better preparation for MRCP(UK) than at others, perhaps because of differences in course emphasis or focus, so that candidates subsequently perform better at the MRCP(UK). If career preferences and pre-admission qualifications cannot explain all of the differences between medical schools, then a reasonableconclusion is that that medical schools also differ in the quality of their training in general medicine. Some schools may therefore be adding more value to their students than others, in relation to taking the MRCP(UK), even taking into account differences in pre-admission qualifications. However, it is of interest that none of the teaching-related measures in the Guardian compilations correlate with MRCP(UK) performance.

The MRCP(UK) examinations are typically taken early in the career, The impact of university teaching on performance is supported by our finding that recency of graduation is a predictor of performance in all three parts of the examination. The coefficient of variation for medical school differences was largest for Part 1 and smallest for PACES, suggesting that postgraduate education dilutes the effects of undergraduate training as time passes. Understanding the mechanisms by which medical school teaching might affect postgraduate examination performance requires more background information than we have available. It is interesting that when a university's students are more likely to report that the teaching of medicine is 'very interesting', then graduates subsequently perform better at MRCP(UK). However, that effect does seem to be secondary to pre-admission qualifications, with students from schools with higher pre-admission qualifications also reporting the teaching of medicine to be more interesting. Teaching can be affected not only by the activities of teachers and students, but also by the environment and institutions in which teaching occurs. A case of particular interest is London, the only university for which there is a specific underperformance of graduates on PACES, the clinical examination of MRCP(UK), and London's medical schools have undergonerepeated reorganizations over the past two decades, which might in part explain the effects on clinical teaching. As the data are aggregated for all London schools, this is difficult to explore further here. An additional confoundingissue for all schools of medicine is the constantchange in curricula. However, our additional analysis of Part1 data goingback to those taking the exam in 1989 (who would have entered medical school in about 1982) shows that the broad pattern of results we have found is long-standing, and therefore could only partly be explained by the changes in medical education initiated by the GMC in Tomorrow's Doctors in 1993 [27]. A detailed examination of individual medical schools (see Figures S11a-11e in additional file 1) shows that for many schools there has been little variation in relative performance between 1989 and 2005. Problem-based learning, introduced in Glasgow, Liverpool and Manchester, has had little obvious impact in the latter two schools, although performance did increase in Glasgow. Despite many, much criticised reorganizations in London, performance overall has improved. Oxford and Cambridge both showedsuddenincreases in performance in the late 1990s, as did Wales. Other schools showed fluctuations, but the overwhelmingimpression is of constancyrather than change, suggesting that curricular and other changes have had little impact on relative performance of schools.

The earlier GMC report of June 2006, Strategic Options for Undergraduate Medical Education [1], had also included a discussion on the potential need to introduce a national medical assessment to ensure that all UK medical graduates have attained an agreedminimum standard of competence. However, the report also highlighted the very limited evidence that existed to support the contention that significant differences in ability existed between graduates of different UK universities. However, an absence of evidence is not evidence of absence, and there are many reasons to believe that schools might differ [28]; a study in the US, for instance, found that graduates of different medical schools differed in their likelihood of malpracticeclaims [29]. We believe that our data provide a primafaciecase that differences in performance exist between UK medical schools, and thus support the case for the routinecollection and audit of performance data of UK medical graduates at all postgraduate examinations, as well as the introduction of a national licensing examination.

The medical information provided on this website is of a general nature and can not substitute for the advice of a medical professional
(for example, a qualified doctor/physician)! Information from the internet could and should NOT be used to offer or render a medical opinion or otherwise
engage in the practice of medicine.