Paths to validating next-gen sequencing assays

Kim Scott

July 2015—As more clinical laboratories tread the unfamiliar ground of next-generation sequencing, they are faced with the age-old challenges of establishing validation and quality control processes. Two experts tackled the topic of molecular QC during a recent CAP TODAY webinar presented in cooperation with Horizon Diagnostics and available for viewing on demand at www.captodayonline.com/cap-today-hosted-webinars/#horizon.

Because the Food and Drug Administration classifies most sequencers as research instruments, the majority of clinical NGS tests require full validation, not just verification. When validating a test, laboratories must compare a number of parameters, including accuracy, trueness, precision, reproducibility, and robustness, according to the CAP’s recommended principles and practices for validating clinical molecular pathology tests (Jennings L, et al. Arch Pathol Lab Med. 2009;
133[5]:743–755). Of these, accuracy is perhaps the most important, said Josh Deignan, PhD, associate director of the UCLA molecular diagnostics laboratories.

“How a lab approaches accuracy for a next-generation sequencing test really sets the stage for how to address all of the other relevant parameters,” Dr. Deignan said.

One approach to validating the accuracy of an NGS test is to compare test results with those of another clinical laboratory performing the same test and whose results are presumed to represent the gold standard.

When performing validations for single gene and small variant panels, a lab typically would require multiple (at least 20) positive and negative samples, with each positive sample containing at least one clinically relevant variant. While it is possible to compare many variants from a single sample, multiple samples are still recommended due to the inherent matrix variability in different extractions and the complexity of other genomic alterations that may exist in specific examples, Dr. Deignan said.

The problem with comparing variants from a single sample, he explained, is other laboratories may not be performing the exact same test that your lab is, which makes it difficult to compare NGS data from the same type of test between two clinical laboratories.

A second approach to validating the accuracy of an NGS test is to compare NGS results with the results obtained from a gold standard method. While Sanger sequencing once was considered the gold standard, Dr. Deignan noted there are cases where a Sanger test is negative while clinical exome sequencing is positive for a particular variant.

Sanger sequencing is no longer the gold standard, he said, because there is potential for allele dropout due to polymorphic positions under primers or unknown heterozygous deletions. When this happens, the sequencing may either miss variants or may erroneously assign homozygosity to a heterozygous/hemizygous variant. What’s more, Sanger sequencing can only detect a minimum allele frequency of 15 percent to 20 percent.

“The conclusion is there may not be a different gold standard method to compare next-generation sequencing to anymore,” Dr. Deignan said. “I would argue that next-generation sequencing is the new gold standard.”

A third approach to validating the accuracy of a next-gen test is to compare the NGS results with the results from well-characterized reference material. A new consortium called Genome in a Bottle is developing the technical infrastructure, including reference standards, methods, and data, to enable translation of whole human genome sequencing to clinical practices. The National Institute of Standards and Technology, in April, made available for purchase its first reference material—8398, human DNA for whole-genome variant assessment. Additional reference materials for sequencing are under development.

One of the QC challenges of NGS is whether to run positive control and no-template control samples with every exome run. This is expensive; it also can be difficult to define a “positive” exome or genome since there are so many variants. Dr. Deignan noted that Sanger sequencing tests don’t require separate positive controls if all of the peaks in the sequence match the sequence of the gene.

Dr. Deignan

“If you think about treating genomic DNA as its own next-gen sample type, which is something that a lot of clinical laboratories are now subscribing to, to me it makes a lot more sense,” he said.

Dr. Deignan does recommend that laboratories validate an NGS assay using their own routine extraction method. However, there may be instances in which other institutions wish to send your laboratory existing extracted DNA for testing instead of fresh blood or cells. Because it’s not possible to validate all DNA extraction methods, one approach is for the laboratory to develop defined QC criteria that confirm sequencing was acceptable. Another approach is to use Sanger sequencing for confirmation, but Dr. Deignan believes this is not the best approach.

“What we did [at UCLA] for the first year and a half of offering clinical exome sequencing was to confirm all reported variants,” he explained. “After that time, we essentially validated our ability to do away with Sanger confirmation for high-quality single nucleotide variants.”

The UCLA molecular diagnostics laboratories still use Sanger confirmation for all low-quality single nucleotide variants and all insertions and deletions. Reads for any variants not confirmed are manually evaluated. The quality score cutoff that UCLA uses is specific to its laboratory; each laboratory will need to develop its own cutoff.

The UCLA laboratory does not confirm somatic NGS panels, mainly because of the challenges in finding a confirmatory method with comparable sensitivity. The CAP leaves it up to individual laboratory directors to decide whether or not to confirm testing.

“I encourage everyone to decide for themselves, using a well-thought-out, data-driven process, how they want to approach this,” he advised. “Next-generation sequencing is the new gold standard for clinical molecular diagnostic testing, and while NGS validation and quality control is challenging and may require slightly different approaches, the principles are the same.”

Dr. Corless

Knight Diagnostic Laboratories in Oregon is a good example of a laboratory that has successfully developed an NGS testing program and is performing comprehensive validations. Knight first got involved with next-generation sequencing in 2012, primarily to better target therapeutics, said Christopher Corless, MD, PhD, Knight’s director and chief medical officer and a professor of pathology at Oregon Health and Science University.

Knight has developed a number of custom amplicon-based panels validated for DNA or RNA from FFPE tissue (see “GeneTrails next-gen sequencing tests”).

“These are relatively small panels ranging from 20 to 76 genes because we have chosen to focus on those genes that we think are truly actionable with drugs today,” Dr. Corless said.

Run 40–50 samples with known SNVs, including insertions/deletions and/or copy number alterations. The laboratory either uses control cell line samples or FFPE tumor samples with known mutations based on Sanger, MassArray, or other assay. Increasingly, the laboratory is cross-validating from one panel to another using tumor samples with mutations at low mutant allele frequency.

Run dilutions for limit of detection.

Perform reproducibility runs.

When assessing a new panel, Dr. Corless recommends considering the following: Are there areas with low coverage, and are known mutation hotspots within these regions? What is the lower limit of detection and how does DNA deamination affect this? What is the size range of insertions and deletions that can be detected?

Another critical factor in NGS testing on tumor samples is assessing the material that is being tested. At Knight, the lower limit of an acceptable sample is 20 percent tumor content. “As a general rule, pathologists tend to overestimate how much tumor they have in the starting material, and this is something we need to keep in mind,” Dr. Corless said.
The amount of tumor content in a sample is especially important when detecting copy number alterations, which are important in precision medicine. Knight has developed an algorithm to detect CNAs in amplicon-based sequencing data. The laboratory used samples with a known FISH status and microarray results to validate the results of the algorithm.

Ultimately, analytical validation of cancer panels should include normal samples, samples with a wide range of mutations, and samples with low mutant allele frequency, Dr. Corless advised. It is important to correlate sequencing results with tumor input, he said. Copy number alteration can be detected, but there are limitations. RNA sequencing also can be useful in detecting gene fusions.

Knight Diagnostic Laboratories has invested a lot to develop its own genomic database, but also relies on public resources such as My Cancer Genome (www.mycancergenome.org), along with the databases from MD Anderson Cancer Center (https://pct.mdanderson.org) and Washington University in St. Louis (https://civic.genome.wustl.edu).

Any laboratory venturing into NGS testing will need to have a robust genomic database, Dr. Corless noted. A laboratory might start with one that is commercially available and then adapt it to its own setting, or may invest in developing one of its own as Knight has done.