1. Summary

External Quality Assessment (EQA) in Cytogenetics has been available in some countries such as the UK for more than twenty years and yet, in other parts of Europe for example France and Czech Republic, schemes are just beginning. A novel approach to EQA using the internet, funded by UK NEQAS and EuroGentest (EUGT), mimics the diagnostic situation so that multiple tests can be requested and EQA cases can be ‘tailor made’ to address a specific chromosome syndrome, disease, or clinical dilemma. The web-based EQA system was shown to be feasible when trialled on a large National EQA scheme (UK). It has also been used to implement a new European EQA scheme, CEQA, set up with the intention of providing laboratories in countries without access to a local EQA scheme the opportunity of participation in EQA. This electronic tool will enable the CEQA scheme to incrementally expand the selection and submission part of EQA without increasing the scheme organiser’s workload.

The internet based EQA overcomes submission delays due to international surface mail. There is also a reduction in administration and assessors’ time compared to a retrospective EQA involving the submission of unique cases for EQA assessment, as participants analyse the same three internet based EQA cases simultaneously. Poor performance was detected in CEQA and UK NEQAS Constitutional EQA schemes and in the UK NEQAS Oncology EQA scheme.

Overall, internet based EQA allows for a varied EQA programme provided the ongoing costs can be financed through the subscription fees. Many EU25 laboratories still do not participate in their National EQA schemes so until EQA participation becomes mandatory as a component of compulsory laboratory accreditation, the future uptake of EQA is unpredictable.

EQA in cytogenetics is generally undertaken in one of two ways: with retrospective assessment the EQA scheme requests material and documentation from specific types of diagnostic case analysed by the laboratory. This form of assessment is valuable insofar as it examines the real practices of the laboratory, but has the disadvantage that interlaboratory comparisons may not be possible because of differences in the complexity of cases submitted. Prospective assessment provides distribution of a consistent batch of material so that inter-laboratory comparison can be made dependably; the disadvantage of this approach is that EQA cases may be given special treatment or priority and not reflect the real practices of the laboratory. As it is difficult to source sufficient appropriate tissue, the material distributed more often comprises slide preparations, images or case scenarios.

The first cytogenetics EQA scheme in Europe was the UK pilot in 1981, which has developed into the present comprehensive scheme. The pilot initially involved a retrospective audit of report times and success rates, before expanding to include a retrospective audit of cytogenetic cases. More recently the UK scheme has introduced prospective assessment with the distribution of case scenarios and diagnostic slides for analysis and interpretation, and currently includes both retrospective and prospective components. Assessment includes scrutiny of technical preparation quality (retrospective only) and appropriateness of tests undertaken, interpretation of abnormalities, significance of the abnormality to the patient and family as revealed by the composition of the report, and use of standard nomenclature.

There are now eight National EQA schemes and one European EQA scheme in Europe. With the exception of Italy and the UK, most National schemes have a separate scheme organiser and management structure for Constitutional and Haematological (or Oncology) Cytogenetics EQA schemes4. The German constitutional scheme started in 1989, basing its structure and EQA format on the UK scheme. This constitutional scheme later expanded to include FISH EQA. Finland and Spain started EQA schemes for constitutional cytogenetics in 1994 and 1998. More recent schemes include the National Italian scheme, the constitutional Dutch scheme, the haematological Spanish scheme and the French and German haematological schemes4 . Poland and France started a haematological and constitutional EQA scheme respectively last year (2006). Finally a European EQA, CEQA was also piloted in 2006 based on the UK NEQAS web-based scheme.

Some National schemes use a prospective form of EQA, either by sending images, or slides. Only the Finnish and French Haematological schemes, each embracing fewer than 25 laboratories distribute tissue samples or cell suspensions. For larger schemes, access to the volumes of blood or cell suspensions necessary is not practical due to the limited source of appropriate material. Consequently most European EQA schemes undertake retrospective assessment of images or slides from cases reported by the participating laboratories.

The web-based EQA provides images for analysis which can be evaluated on-line or downloaded to an image analysis system. This EQA has been developed to mimic the diagnostic situation closely and overcomes some of the limitations of other forms of EQA in that it allows the participant to choose appropriate follow-up tests (for example, FISH) to elucidate the chromosome abnormality. This is a major improvement on conventional distribution of slides or images where the inclusion of additional test material inevitably reveals the answer.

This document describes the internet based EQA system, some of the problems encountered, the different approaches to the on-line EQA taken by laboratories and a summary of the recent EQA results. While this approach to EQA is feasible, there are maintenance costs which have to be recovered through laboratory subscriptions. However, the initial costs to develop and implement this form of EQA are likely to be too prohibitive for small voluntary-contribution EQA schemes unless external funding can be obtained.

Results and discussion

3.1 Internet based EQA system

The internet based EQA mimics the diagnostic service such that cases can be viewed on-line and additional cytogenetic investigations can be chosen. The EQA cases can be tailored to a specific clinical scenario with the option of multiple additional investigations e.g. metaphase images; parental bloods; FISH investigations; additional banding techniques; previous family or cytogenetic history. An example of the multiple options is shown below in Fig.1.

FIGURE 1 - Diagrammatic representation of the multiple options.

The web-based EQA system is password protected and passwords are allocated by the administrator and can be given to a laboratory or an individual participant. The website has been designed for easy participant navigation. Each case has menu bars to select the various test categories (Fig 2, red arrow ), a visible route pathway to aid navigation of the website (Fig 2, blue arrow ) and it is possible to return to the request card or report the case at any stage (Fig. 2, black arrow) The first screen seen by the participant shows the referral card that gives the patient clinical details. The participant then selects the appropriate tests from the menu bar to analyse images, select further tests as necessary to supplement their analysis, and ascertain the result based on the clinical information given. Within each test category there may be a series of cascading options or sub-categories e.g. FISH may have subcategories such as "microdeletions"; "chromosome-specific"; "dual fusion"

probes” etc. Under the sub-category “microdeletions”, the participant can then select the appropriate specific probes needed e.g. DiGeorge syndrome or Miller Dieker syndrome. Once a specific test has been selected, the participant will view either a series of images as thumbnails, or a table or diagram, or some text depending on the type of test requested. It is possible for the participant to magnify the thumbnails to fit the monitor screen to aid visualisation and analysis.

The web-page is interactive, so the participant can make comments as each test or image is reviewed, as well as being able to track which images they have already viewed or analysed by clicking the appropriate button on the screen. All comments are visible to the participant and can be amended at any time until completion of the case. The software gives each image a unique number and enables the participant to colour-code the periphery of the image so that when logging-on at a later stage it is immediately clear which images have been previously analysed or just viewed (Fig. 2). This also has the advantage to the checker or verifying analyst when the case is reviewed prior to submission.

The participant can write a draft report at any stage of the analysis to be available when they next enter the website, provided it is saved. Once the case is analysed and the checking procedure completed, the final report can be submitted by entering a second password (exit code). The exit code ensures against any accidental submission before the case is complete, but once submitted, the case cannot be revisited for revision.

Once the assessment process is complete, participants can revisit the EQA case for post submission review, but cannot change their submission.

3.2 Scheme Administration

The internet EQA is easy for the administrator to manage as each case can be individually created and the IT tools allow total flexibility to add or delete investigations, images or graphics. The images can be stored in the database as TIFF or JPEG files. In addition it is possible to duplicate a previous EQA case or create a template case which can then be modified to create a new EQA case scenario, reducing the administration for the scheme organiser. Each investigation or test can be given a ‘cost’ or weighting and the software calculates the total cost incurred by the participant when selecting the different tests. A participant may select the same test several times to visualise the results but only incurs a ‘cost’ on the initial visit to the webpage. Participants attempting to select every test to obtain the result are thereby distinguishable from those participants that obtain the answer using the appropriate tests to ascertain the chromosome abnormality.

The system is flexible to allow the creation of different EQA scenarios, and cases are given specific start and finish dates. The administrator assigns participants to specific cases, so that participants not registered for a particular EQA will not get access. Although the administrator can view a case at all times, participants can only contribute and revise between the specified start and finish dates. Once the finish date has passed, participants who have completed their submission can view the case for educational purposes but not make any changes, while assessors can only access the participant’s submission once the case is closed. Assessors can be assigned to all cases or to specific cases, to avoid assessing their own laboratory and to exploit their particular areas of expertise. The identity of the laboratories is coded by the Scheme Organiser, the sole code-holder, and not divulged to the assessors. Assessors can visualise the report and any comments submitted by the participant and, if required, examine a map of the participant’s web-page usage (Fig 3 & 4). The IT system audits website usage so it is possible for the administrator and assessors to visualise diagrammatically the webpage usage for each case made by each participant. Excessive web-page usage indicates that the participant

FIGURE 3 - Audit trail with a unit cost 20

has chosen inappropriate tests that can be commented upon in their individual laboratory EQA reports. The web-page usage can be allocated a ‘cost’ for unnecessary additional tests requested by the laboratory and a penalty allocated if appropriate (Fig 4). This audit trail is of particular use when the incorrect diagnosis is obtained as it may be possible to ascertain where in the decision making process the analysis or interpretation of the abnormality was incorrect.

FIGURE 4 - Audit trail with a unit cost 274

3.3 Modifications to the system required following piloting of EQA scheme.

Initial teething problems were encountered when the on-line EQA was piloted and these software ‘bugs’ were corrected. Further modifications made after feedback from a participants questionnaire asking them to comment on their experience of on-line EQA process and how the on-line EQA can be improved. A few participants (4/80) had older versions of Windows Explorer internet browser that did not support the software and these laboratories had to ask their IT departments to upgrade some of their computer software packages. In addition, a few institutions found that their servers cached web-pages so that any comments made or changes to the report were not immediately visible. Participants were informed how to overcome this problem. The ‘session time out’ for individual web-pages had to be lengthened as it was found that many laboratories did the analysis from the screen which could taking up to an hour and did not download the images into an image capture system. In addition, as participants chose to analyse on-line, the images from the first EQAs were too large and in subsequent EQAs the images were reduced in size to fit the screen.

Improvements to the overall EQA design were also incorporated. URL links to manufacturers FISH (fluorescent in situ hybridisation) probes have been incorporated onto the web-pages to assist laboratories in identifying the exact location of the probes. As more EQAs were completed on-line, it became clear that some form of case management was required. Consequently another tier was introduced to the software so that cases belong to a specific EQA which in turn belong to either the Constitutional scheme or the Oncological Scheme.

As each EQA case can have different investigations available, to improve web friendliness and make navigation easier, a diagrammatic representation of all the tests, each with a web link to the specific webpage, is now available on the menu bar (Fig. 5).

Finally, 20% of participants initially had problems accessing the website because of incorrect password or user name entry. Participants had to be reminded not to introduce any spaces and ensure the correct letter case was used when entering the password. Simpler passwords have since been introduced where there is no random mix of upper and lower case letters. On three occasions the problem was not related to password entry and the developer had to reboot the software programme because of technical problems arising from maintenance carried out by the server provider.

3.4 Participant usage of the website

The time period for which an EQA case is made available for analysis corresponds to European cytogenetic report times for diagnostic results5. Participants usually had 3-4 weeks to analyse and report a case on-line. In 2005 the peak usage of the website occurred in March, April and July- September when EQA cases went live (Fig.6). If three EQA cases were put on-line simultaneously, participants did not stagger their analysis but analysed all three cases at once. Further breakdown of the usage during September showed that peak usage was the last few days before the closing date when all three cases were on-line (Fig. 7). As the main load on the server was when an EQA case was about to close, UK NEQAS decided to stagger the start and close dates of the cases to spread the load over a six week period. The website creates a log of each entry a participant makes and it is possible to see when a laboratory first examines an EQA case and when the final report is submitted. Using these start and end dates it is possible to visualise how participants approached the EQA case. Most participants viewed a case within the first week of it opening but the final

FIGURE 6 - On-line usage during 2005

FIGURE 7 - On-line usage during September 2005.

reporting of a case was left by the majority to the last three days (Fig. 8 & 9). Apart from case 1 (Fig. 8) participants had 28 days to complete a case on-line (Fig. 8-10 show the individual blood EQA cases in 2006 and Fig. 11 the combines the data from all three blood EQA cases). In all four graphs there are regular periods of low activity that correspond to the weekends.

FIGURE 13 - Postnatal bloods- length of time taken to report an EQA case.

the case earlier. The only participants that completed the EQA case in a less than seven days were those that started the EQA case in the last week (Fig 13). The majority of participants took two weeks to report the EQA case on line. Staggering the start and close dates of the three blood EQA cases in 2006 spread the load on the website over the period the EQA cases were open (compare Fig. 7 with Fig. 14).

FIGURE 14 - Postnatal bloods- load on the website with staggered dates for the EQA cases.

Participants also fell into two basic categories, those that analysed every test available on the website and those that were more discriminating in the tests they selected. While the majority of participants ascertained the

correct answer, not all made efficient use of their time and resources if this webpage usage reflects their diagnostic practice. The “cost” allocation for each investigation was introduced in 2006 and a wide variation in costs was found (see blood EQA, Fig. 15). The costs ranged from ten units to 764 units, a 70 fold variation. The case involved the diagnosis of a blood sample with balanced insertion, 46,XY,ins(18;8)(q11.2;q13q24.1) and the expected answer could have been obtained within a maximum cost of 42 units assuming parental bloods (20 units) and confirmatory FISH studies (14 units) were undertaken. The modal cost for this particular blood EQA was between 40 & 60 units with a mean cost of 153.8 (Fig 16). While 27/42 (64%) participants incurred a cost of <100 units, 13/42 (25%) of participants incurred costs greater than 200 units with two of these participants incurring unit costs > 730.

Participants were required to submit a report on line including the correct nomenclature according to ISCN 20056 to describe the chromosome rearrangement, a written description and clinical interpretation of the abnormality. There was a very wide variation in the content of submitted reports, with some participants not explaining or interpreting their findings. For instance few laboratories related the chromosomal findings to the presenting clinical features or discussed the significance of the result. Some participants had difficulty reporting a normal result succinctly and suggested inappropriate and costly investigations to ensure nothing had been missed. The variation in reporting styles was unexpected, both within and between countries (see Appendix A for some examples for a normal blood male result). The introduction of the European Cytogenetic Guidelines5 may resultin further harmonisation in reporting over time where National laws allow. In some European countries (e.g. Greece), the cytogenetic laboratories are only able to report the analysis: by law the interpretation is undertaken by the referring clinician. Elsewhere the interpretation is made by the laboratory although in some European countries the head of the laboratory also has to be clinically qualified. Where the interpretation is made by a clinician outside the laboratory, these practitioners are now being encouraged to be involved in the EQA process so the complete cytogenetic result is assessed for EQA.

3.5 EQA results

Participants were assessed for their analytical and interpretative performance according to previously set performance criteria4,7. The EQA involves a selection of cases that are either normal or abnormal or a mixture of both. The on-line EQA was initially piloted and tested on postnatal blood cases and then prenatal cytogenetics (both Constitutional EQAs) before expanding to include Oncology.

Five pilot EQAs involving eight EQA cases were trialled on the internet system for UK NEQAS and CEQA schemes (Table 1). A total of 80 laboratories from 24 countries participated from across the world. Pilot EQAs are voluntary and 14% of laboratories failed to submit their report before the deadline.

Table 1

EQA

No. Cases

No. Participants

No. Completed

Performance of completed assessments

2005

Amniotic Fluid FISH

1

34

20

Not applicable

Oncology (ALL FISH)

1

44

41

Not applicable

Oncology B-cell LPD

2

13

13

Not applicable

2006

Oncology B-cell LPD

2

20

19

All satisfactory

CEQA

2

25

19

5 poor performances

Since these pilots UK NEQAS has completed a further seven EQA involving 14 EQA cases for Constitutional and Oncology Cytogenetics (Table 2). A total of 82 laboratories from 20 countries participated from across the five continents (Americas, Asia, Australasia, Africa and Europe) and of these Europe was the largest contingent with 12 different countries participating.

Table 2

EQA

No. of Cases

No. Participants

No. Completed

Performance

2005

Blood

3

44

42

4 poor performances

Constitutional FISH

1

34

32

All satisfactory

Oncology AML FISH

1

43

38

1 poor performance

2006

Blood

3

46

43

2 poor performances

Constitutional FISH

1

36

64

All satisfactory

CVS

2

38

36

2 poor performance

Oncology inc. FISH

3

44

43

6 poor performances

Significant differences in reporting of analysis and interpretation of the normal and abnormal results was evident. All participants correctly analysed the normal results and mosaics but inappropriate advice was given in a

Table 3

EQA

Case

Result

Outcome

2005

Blood

1

46,XX,del(2)(q37.1)

Abnormal

Blood

2

45,X[24]/46,X,i(X)(q10)[6]

Abnormal

Blood

3

46,XY

Normal

FISH

1

Two signals for ELN*

Normal

Oncology FISH

1

BCR/ABL rearrangement (indicative of an ALL relapse)

Abnormal

2006

Blood

1

46,XY,ins(18;8)(q11.2;q13q24.1).

Abnormal

Blood

2

46,XY,t(3;13)(q12;q12)

Abnormal

Blood

3

46,XY in a phenotypic female

Abnormal

FISH

ish del(17)(p11.2p11.2) (Smith Magenis syndrome)

Abnormal

CVS

1

Trisomy 13 on direct, normal result on culture

Normal**

CVS

2

Two cell lines. Normal male karyotype and abnormal male karyotype with trisomy 15 due to a der(15;15)(q10;q10)

Normal**

Oncology

1

inv(16)(p13q22) and t(9;22)(q34;q11.2)

Abnormal

Oncology

2

MLL (11q23) fusion positive rearrangement

Abnormal

Oncology FISH

3

PML/RARA insertion

Abnormal

* a single signal would be indicative of Williams syndrome

** CPM- Confined placental mosaicism - Likely outcome normal

small proportion of cases. One unexpected finding was the reporting of a 46,XX,del(2)(q37.1) blood case as a normal female (46,XX) by three participants (Table 3). This may indicate a poor level of analysis or unfamiliarity of synchronised blood cultures that produce longer metaphases and enable the analyst to visualise the chromosomes at a higher resolution. Slides from three cases were sent to these laboratories to follow up the poor performance and one laboratory still failed to detect a terminal deletion in the long arm of chromosome 2, 46,XX,del(2)(q37.1).

Identification of poor performance is an essential part of EQA and enables a laboratory to address the internal quality issues and improve the diagnostic service offered. Unfortunately, over time, UK NEQAS has found that a small percentage (2/82 laboratories to date) of the laboratories fail to register for EQA the following year. On each occasion the laboratory has had a poor performance within the first year of participating suggesting they prefer to ignore the problem rather than address it.

3.6 Further Applications of the software

3.6.1. CPD

While this software programme has been used for internet based EQA, its flexibility allows for other applications such as Continual Professional Development (CPD), Competency testing or Independent learning programmes. The website can also be adapted for Educational purposes to give rare case scenarios.

3.6.2. CEQA

CEQA was successfully piloted using the internet based EQA in 2006. Twenty five participants enrolled and 19 laboratories from 15 countries submitted a report. Seven laboratories had never participated in EQA before and successfully navigated the web-pages to attain the correct answer. It is impossible to predict how quickly the scheme will expand or how many participants there will be for the CEQA scheme in the next three years. The estimated number of participants, given in Table 4, is based on the growth of EMQN (European Molecular Genetics Quality Network) from 1985 to present day (personal communication, Simon Patton).

Table 4

Year

1

2

3

4

EMQN

14

50

75

125

No. EQA offered

1

3

4

5

Mean no. labs per EQA

13

17

19

29

CEQA Participants

20

50

80

100

EQA work package WP1.4 identified 738 cytogenetic laboratories in Europe and of these approximately 350 participate in existing EQA schemes. There are therefore potentially 400 laboratories without access to, or choosing not to, participate in EQA. The experience of EMQN has been that many laboratories choose to participate in both EMQN and their National scheme, so there potentially could be 600+ participants for CEQA in the longer term, especially as accreditation of cytogenetic laboratories becomes a requirement1,2,3.

As most National EQA schemes are currently free or charge a nominal fee, the introduction of charges to participate in CEQA will have to be gradual in order to accustom participants to pay for EQA. Only with the introduction of fees will CEQA be sustainable in the long term.

All but two National schemes are run on a voluntary basis, jeopardising sustainability as schemes are susceptible to closure or interruption when the scheme organiser retires or moves from their diagnostic cytogenetic post. CEQA has been established with a part time scheme co-ordinator and quality manager. The baseline costs to set up a National or European based Cytogenetics EQA scheme are the same whether the EQA involves a paper exercise, CD-ROM or an internet based system. However, the additional cost incurred by developing an internet based EQA are considerable and could not have occurred without external funding. With an increasing number of participants, the cost of the CEQA scheme to each laboratory will reduce as the hosting and support costs are a fixed cost.

4. Conclusions.

The on-line internet based EQA has provided a varied EQA programme for both a National and European EQA scheme, covering Oncology and/or Constitutional Cytogenetics. The EQAs from the UK NEQAS scheme and CEQA have demonstrated that this form of EQA is able to detect poor performance. The vast majority of laboratories obtain a satisfactory performance in all their EQA rounds giving assurance to patients and clinicians that the laboratory is providing a quality service with reliable and accurate results.

The software was developed by Waypoint Systems. This web based EQA system was funded through CPA (UK) Ltd, UK NEQAS for Clinical Cytogenetics and EuroGentest. This paper was completed with the support of EuroGentest (grant no: LSHB-CT-2004-512148).

Appendix A

46,XY Cytogenetic diagnosis: 46,XY Normal cytogenetic male. The karyotype done to Baby X is normal, as his parent’s karyotype. Also the FISH tests done are normal, including Prader-Willi syndrome. This rules out cytogenetic syndromes due to chromosome number, coarse structural chromosome disorders microdeletions syndromes and disorders in sexual chromosomes. Molecular studies of uniparental disomy are normal. One of the possible diagnoses of floppy baby is Prader-Willi syndrome. Once FISH (for Prader-Willi) and UPD tests are normal, methylation imprinting test should be done to rule out this syndrome with 99% of confidence. Other molecular tests could be done to try to reach an etiologic diagnosis. A thorough clinical history, family kinship and family clinical background should be gathered to guide the rest of the tests that must be done (NMI, EMG, blood test, enzymatic tests etc... Other molecular tests could be done to try to reach an etiologic diagnosis always taking into account the results given before.

46,XY Apparently normal male karyotype.

46,XY Chromosome analysis of the blood sample from this patient has shown an apparently normal male karyotype. Parental blood samples from this patient have also been analysed and shown to have apparently normal chromosome Karyotypes, please refer to a copy of the reports (x-ref). Molecular Cytogenetic Result. Fluorescence in situ hybridisation studies have been carried out on this patient using the Vysis probe for the UBE3A locus on chromosome 15; the results have demonstrated NO DELETION of the UBE3A region, which should exclude approximately 70% of PWS/AS cases; however further exclusion would require FISH testing using the probe for the SNRPN region. Molecular Genetic studies have also been carried out on this patient which show bi-parental inheritance of chromosome 15 thus excluding Uniparental Disomy 15. If a diagnosis of PWS/AS is still suspected, we would recommend that this patient is referred to the Clinical Genetics Department for further assessment which may require additional molecular tests to rule out mutations. There was no deletion of the LIS1 gene on 17p13.3, which should exclude a diagnosis of Miller Dieker Syndrome. Additional FiSH tests were also carried out which included centomere probes, M-FISH, whole chromosome paints, and telomere specifc probes, all of which showed a NORMAL signal pattern.