Abstract

Aim To conduct a methods correlation study of three different assays for the detection of mutations at EGFR gene in human formalin-fixed paraffin-embedded tumour (FFPET) specimens of non-small cell lung carcinomas (NSCLC).

Methods We conducted a 2-site method comparison study of two european conformity (CE) in vitro diagnostic (IVD)-marked assays, the
cobas EGFR Mutation Test and the Therascreen EGFR29 Mutation Kit, and 2× bidirectional Sanger sequencing. We blind-tested
124 NSCLC FFPET specimens with all three methods; the cobas test was performed at both sites. Positive (PPA) and negative
percent agreements (NPA) were determined for the cobas test versus each of the other two methods. Specimens yielding discordant
test results between methods were further tested using quantitative massively parallel pyrosequencing (MPP).

Results PPA between cobas and Sanger was 98.8%; NPA was 79.3%. Overall there were seven discordant results. MPP confirmed an exon
19 deletion in two cases and L858R mutation in four cases. PPA between cobas and Therascreen was 98.9% and NPA was 100%. There
was one discordant result. Reproducibility of the cobas test between the two sites was 99.2%.

Conclusions The invalid rates for the cobas test and Therascreen were lower than Sanger sequencing. The cobas and Therascreen assays
showed a high degree of concordance, and both were more sensitive for the detection of exon 19 deletion and L858R mutations
than Sanger. The cobas test was highly reproducible between the two testing sites, used the least amount of DNA input and
was the only test with automated results reporting.

Introduction

Anti-EGFR tyrosine kinase inhibitors (TKIs) are indicated for first-line therapy in patients with EGFR mutation positive non-small cell lung carcinomas (NSCLC).1–6 In Europe, the European Medicines Agency labelling states that gefitinib and erlotinib are indicated for patients with locally
advanced or metastatic NSCLC with activating mutations of EGFR. In the USA, the Food and Drug Administration has not yet approved the use of erlotinib as first-line therapy, though EGFR testing is recommended in the NSCLC guidelines from major US oncology organisations, such as the National Comprehensive Cancer
Network and the American Society of Clinical Oncology to guide treatment regime. No guidance recommending specific testing
methods or assay attributes currently exists. Approximately 90% of all activating mutations are exon 19 deletions and a point
mutation (L858R) in exon 21.7 At present, although it is recommended that the presence of activating mutations in the EGFR gene should be identified before treatment with gefitinib or erlotinib, there is no consensus regarding the specific mutations
that should be tested nor methods that should be used.8 A number of sequencing and PCR-based methods for detecting EGFR mutations are currently in clinical use; however, it is not clear which technique offers the best performance in terms of
sensitivity, specificity, reproducibility and accuracy.9,10 Additional attributes that are desirable for EGFR tests include short turnaround time; low input DNA requirement, as large specimens are often difficult to obtain; test robustness,
so that results are not influenced by pathological factors, for instance, varying tumour content, lymphocyte infiltration,
tissue necrosis, and mucin content; and automated reporting, which can reduce variability in the interpretation of results.

We performed a two-centre study to compare the analytic performance and workflow characteristics of the cobas EGFR Mutation
Test against two other methods commonly used in the clinical setting: 2× bidirectional Sanger sequencing and the Therascreen
EGFR29 Mutation Kit, using a blinded panel of formalin-fixed paraffin-embedded tumour (FFPET) NSCLC specimens.

Materials and methods

Mutation testing methods

The cobas EGFR Mutation Test kit (‘AS-PCR test’, Allele-Specific PCR test, Roche Molecular Systems, Inc., Branchburg, New Jersey, USA) is a CE-IVD-marked
allele-specific PCR test designed to detect the presence of 41 mutations in exons 18, 19, 20 and 21 of the EGFR gene in NSCLC FFPET specimens. The test requires 150 ng total DNA input, which can typically be obtained by using one 5 μm
FFPET section. All analysis and results reporting is fully automated.

The Therascreen EGFR29 Mutation Kit (‘ARMS test’, Amplification Refractory Mutation System, Qiagen, Manchester, UK) is CE-IVD-marked real-time PCR assay that
combines the ARMS and Scorpion fluorescent primer/probe systems to detect 29 mutations in exons 18, 19, 20 and 21 of the EGFR gene. Reactions were run in the ABI 7500 Real-Time PCR System (Life Technologies, Warrington, UK) and analysed using ABI
7500 software (V.2.0.5). Per the package insert, the ARMS test requires ≥20 ng of amplifiable genomic DNA from FFPET specimens.
According to the laboratory-validated clinical protocol, ∼100 ng of total DNA, as measured by spectrophotometry, is used per
PCR reaction to account for the partial degradation of FFPET DNA without resulting in oversaturation of the reaction. The
test thus requires a total DNA input of 800 ng.

Sanger sequencing (‘Sanger’): Mutation screening for exons 18, 19, 20 and 21 of the EGFR gene was carried out using PCR conditions and 2× bidirectional direct sequencing following previously described protocols.11,12 According to laboratory-validated protocols, 150 ng of DNA is used per PCR performed, for a total of 600 ng total DNA input.

Study design

From a panel of 300 vendor-purchased FFPET NSCLC tumour specimens, 133 specimens were selected for this study based on histologic
and clinical characteristics. All selected specimens were from a cohort of Asian non-smoking women with adenocarcinoma, characteristics
of which have been shown to have higher frequency of EGFR mutation positive lung cancer.14,15 The study design is depicted in figure 1. Of the 133 specimens, 114 specimens were selected at random; 13 specimens were previously characterised using real-time
PCR, and six specimens that were known to contain rare mutations (L861Q, S768I and exon 20 insertions). Testing at both sites
was performed blinded.

The 133 tumour specimens were each partitioned to yield five 5 μm sections per panel member. One section was mounted on a
slide and stained with haematoxylin and eosin, coded and reviewed by two pathologists (FL-R and EC) who assessed the tumour
content, lymphocyte infiltration, extent of necrosis and mucin content. None of the specimens was subjected to macrodissection
in order to avoid bias and intersite variability of DNA specimens, and therefore, truly comparing the analytical capabilities
of each method on the same samples. Eight specimens were excluded due to insufficient tumour content (≤1%), and one specimen
was excluded due to invalid results by all methods, leaving 124 evaluable specimens for methods comparison analysis. Two 5 μm
curl sections per panel member were sent to Clinical Site 1 (Hospital Universitario Madrid Sanchinarro, Madrid, Spain) for
analysis using the AS-PCR test and Sanger, and two 5 μm curl sections per panel member were sent to Clinical Site 2 (Institute
of Cancer Research, Surrey, UK) for analysis using the AS-PCR test and the ARMS test.

DNA for the AS-PCR test was isolated from a single 5 μm section per panel member at each site using the cobas DNA Sample Preparation
Kit (Roche Molecular Systems).16 The DNA eluate was subsequently tested using the AS-PCR test according to the package insert.17 DNA for Sanger sequencing and the ARMS test was isolated from separate single 5 μm sections per tumour panel member using
the QIAamp DNA FFPET tissue kit and automated QIAcube robot (Qiagen, Hilden, Germany). The DNA eluate was then tested with
Sanger according to a standard laboratory protocol at Site 1 and the ARMS test according to the vendor-provided protocol at
Site 2.

Specimens with invalid test results or with discordant results between any of the methods were retested according to manufacturer/procedure
instructions. Criteria for retesting were:

ARMS test: positive controls have not amplified specific product; mixed standard delta Ct is not within specified range; no
template control has Ct<38; any sample with control gene Ct>37–38.

Quantitative MPP was performed on all discordant and invalid specimens.

Study objectives

Specific objectives of the study were to:

Compare analytical performance based on methods correlation parameters of PPA and NPA between the AS-PCR test with the ARMS
test and Sanger sequencing (respectively, at each site)

Assess the reproducibility of the AS-PCR test at two independent laboratories

Assess the frequency of invalid test results for all methods

Assess the effects of pathological factors on the analytical performance of the AS-PCR test

Compare turnaround times between all methods.

Methods correlation

The PPA and NPA of the AS-PCR test were compared with the two other testing methods (Sanger and ARMS) for 124 evaluable FFPET
NSCLC specimens. Agreement analysis for all methods is based on reportable results (table 1). Quantitative MPP was performed for all specimens for which the AS-PCR test and the comparison method gave discordant results
and/or for which one of the two testing methods gave an invalid result. False positive (FP) and false negative (FN) rates
were calculated for both methods using the AS-PCR test as the reference.

Reproducibility

The reproducibility of the AS-PCR test was evaluated by comparing the results at the two independent clinical laboratory sites.
Discrepant analysis was performed using MPP on all specimens for which the AS-PCR test gave discordant results and/or when
an invalid result was obtained.

Invalid test rate

The number of invalid test results from the evaluable tumour panel were recorded and compared across the three testing methods.

Impact of specimen attributes on analytic performance

The following pathological characteristics were assessed and graded according to the criteria:

Tumour content: high (≥50%) vs low (<50%)

Lymphocyte infiltration: high (≥10%) vs low (<10%)

Tumour necrosis: high (≥50%) vs low (<50%)

Mucin content: high (≥50%) vs low (<50%).

Workflow Assessment

Assay turnaround time from DNA isolation to results reporting was compared for all methods, assuming one 8 h shift/day.

Results

Invalid test rate

Of the 248 specimens that were evaluated by the AS-PCR test (124 at each site), two specimens gave initially invalid test
results at both sites, upon retest one specimen remained invalid, resulting in an invalid rate of 0.4% (1/248) for both sites.
Of the 124 specimens evaluated by Sanger, 23 specimens (18.5%) were initially invalid; upon retesting, 4.0% (5/124) remained
invalid. Interestingly, the five samples which remained invalid by Sanger sequencing were characterised by poor quality of
the extracted DNA (ratio A260/A230 ≤1). Of the 124 specimens evaluated by the ARMS test, one specimen was originally invalid and was resolved upon retest. The
proportions of invalids for AS-PCR and ARMS were equivalent.

Methods correlation with Sanger

Of the 124 specimens tested at Clinical Site 1 using the AS-PCR test and Sanger, six specimens were invalid by either the
AS-PCR test or Sanger, and five contained mutations that were not reportable by AS-PCR test (L861Q mutations), leaving 113
evaluable specimens for comparison. The initial agreement analysis showed a PPA of 98.8%, NPA of 79.3%, and an overall percent
agreement (OPA) of 89.8% (table 2).

Seven specimens with discordant test results were subjected to MPP. One specimen reported as ‘mutation not detected’ (MND)
by the AS-PCR test and ‘mutation detected’ (MD) (exon 20 insertion) by Sanger sequencing was reported as MND by MPP. Six specimens
were reported as MD (2 exon 19 deletions and 4 L858R mutations) by the AS-PCR test and MND by Sanger. MPP reported all these
specimens as MD. MPP indicated that Sanger had missed six mutations. Interestingly, most of these samples had a tumour content
≤20%, and were characterised by poor quality of the extracted DNA. Following discrepant resolution with MPP, the PPA was 100%,
NPA was 100%, and OPA was 100% (table 2). The FP and FN rates for Sanger were 1.2% and 20.7%, respectively, using AS-PCR test as the reference.

Methods correlation with the ARMS test

Of the 124 specimens tested at Clinical Site 2 using the AS-PCR test and the ARMS test, three contained mutations that were
not reportable by AS-PCR test (L861Q mutations), leaving 121 evaluable specimens for comparison. ARMS test did not call two
specimens reported by Sanger as L861Q. The initial agreement analysis showed a PPA of 98.9%, NPA of 100% and an OPA of 99.2%
(table 3).

One specimen with a discordant test result, MD L858R mutation with the ARMS test and MND with the AS-PCR test was subjected
to MPP. This specimen was confirmed by MPP to contain an L858R mutation. Following discrepant resolution with MPP, the PPA
was 98.9%, NPA was 100%, and OPA was 99.2% (table 3). The FP and FN rates for ARMS test were 1.1% and 0%, respectively, using AS-PCR test as the reference.

Reproducibility

On the 124 evaluable specimens, one specimen remained invalid upon testing with the AS-PCR test, leaving 123 specimens for
the analysis of AS-PCR test reproducibility between the two sites. Of these, 122/123 (99.2%) produced concordant results.
The one discordant sample contained an L858R mutation according to MPP.

Impact of specimen attributes on analytic performance

Pathologic assessment of the 124 FFPET specimens revealed varying degrees of tumour content, lymphocyte infiltration, necrosis
and mucin content (table 4). Regarding tumour content, it is important to note that mean, median and range were as follows: 59.9%, 60% and 5–95%, respectively.
None of these pathologic characteristics had an observable effect on agreement analysis, reproducibility, or the invalid rate
of the AS-PCR test (data not shown).

Distribution of pathological characteristics in the 124 evaluable specimens

Workflow

Turnaround time per run for the AS-PCR test was approximately 1 day for 24 samples; for Sanger it was approximately 7 days
for 24 samples; and for the ARMS test it was approximately 1 day for 10 samples.

Discussion

The AS-PCR test was highly reproducible (>99%) between sites, a notable attribute given the high variability of other methods
between laboratories.18,19 The high reproducibility may be attributed to the extensive validation studies and automated reporting.20 Given the current concerns around intratumour heterogeneity, minimising variability in results interpretation and achieving
high lab-to-lab reproducibility is critical in ensuring accurate patient diagnosis and treatment.21,22

The AS-PCR test requires 150 ng of total DNA input and detects 41 mutations in exons 18–21. This amount of DNA can be isolated
typically from a single 5 μm section of NSCLC FFPET specimen, a significant assay attribute given the current specimen tissue
constraints and need for effective testing prioritisation. Although the current study only involved surgical specimens, the
AS-PCR test has a similar performance when using small biopsies despite the lower amount of DNA obtained in this setting (D
González de Castro, B Angulo, E Conde and F Lopez-Rios, unpublished data). In patients with advanced lung NSCLCs there is
often not enough sample left for all the predictive biomarkers (eg, EGFR and Anaplastic Lymphoma Kinase (ALK) testing) after routine pathology workout.23,24 In fact, the use of a single section for each of the biomarkers might favour a simultaneous instead of a sequential approach
in this setting.25 The ARMS test requires >160 ng of amplifiable DNA (equivalent to 800 ng of total DNA input in our experience) and detects
29 mutations in exons 18–21. With the Sanger sequencing method, four PCR reactions were performed (one reaction for each of
the four exons) using approximately 150 ng DNA per PCR reaction (600 ng total) to theoretically detect any mutation in these
four exons.

The results of the analysis of discordant results show that the AS-PCR test and the ARMS test are more sensitive than Sanger
for detecting EGFR mutations. The AS-PCR test detects >90% of the activating mutations in exons 19 and 21 according to frequencies based on
COSMIC database V.54, but the tested version of the kit does not detect the L861Q mutation. Based on published data, it is
not possible to conclude whether this mutation is sensitising for erlotinib and gefitinib.26–28

In summary, the low invalid rates for the AS-PCR test and ARMS test allowed both tests to provide results in the majority
of cases. The AS-PCR test and the ARMS test results were highly concordant with one another and highly sensitive. Of the three
methods, the AS-PCR test had the lowest DNA input requirements and was the only test with automated results reporting.

What the paper adds

New targeted therapies are being approved with companion diagnostic tests. Unfortunately, this has not been the case for EGFR inhibitors. Therefore, it is not clear which technique offers the best performance in terms of sensitivity, specificity,
reproducibility and accuracy. We performed a two-centre study to compare the analytic performance and workflow characteristics
of the three most widely used methodologies for EGFR mutation testing.

Take-home messages

The low invalid rates for the AS-PCR test and ARMS test allowed both tests to provide results in the majority of cases.

The AS-PCR test and the ARMS test results were highly concordant with one another and highly sensitive.

Of the three methods, the AS-PCR test had the lowest DNA input requirements and was the only test with automated results reporting.

Acknowledgments

We would like to thank the Sequencing Service at the Spanish National Cancer Centre (CNIO) for performing the direct sequencing
assays. Miller Medical was contracted by Roche Molecular Systems and contributed in the drafting of this manuscript. This
study was sponsored by Roche Molecular Systems, Pleasanton, CA.

Open Access This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0)
license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative
works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/