Multi-Institutional Evaluation of Interrater Agreement of Variant Classification Based on the 2017 Association for Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists Standards and Guidelines for the Interpretation and Reporting of Sequence Variants in Cancer
This multi-institutional study was undertaken to evaluate interrater reliability of the 2017 Association for Molecular Pathology/American Society of Clinical Oncology/College of American Pathologists guidelines for interpretation and reporting of oncology sequence variants and to assess current practices and perceptions surrounding these guidelines. Fifty-one variants were distributed to 20 participants from 10 institutions for classification using the new guidelines. Agreement was assessed using chance-corrected agreement (Cohen κ). κ was 0.35. To evaluate if data sharing could help resolve disagreements, a summary of variant classifications and additional information about each variant were distributed to all participants. κ improved to 0.7 after the original classifications were revised. Participants were invited to take a web-based survey regarding their perceptions of the guidelines. Only 20% (n = 3) of the survey respondents had prior experience with the guidelines in clinical practice. The main perceived barriers to guideline implementation included the complexity of the guidelines, discordance between clinical actionability and pathobiologic relevance, lack of familiarity with the new classifications, and uncertainty when applying criteria to potential germline variants. This study demonstrates noteworthy discordances between pathologists for variant classification in solid tumors when using the 2017 Association for Molecular Pathology/American Society of Clinical Oncology/College of American Pathologists guidelines. These findings highlight potential areas for clarification/refinement before mainstream clinical adoption. This multi-institutional study was undertaken to evaluate interrater reliability of the 2017 Association for Molecular Pathology/American Society of Clinical Oncology/College of American Pathologists guidelines for interpretation and reporting of oncology sequence variants and to assess current practices and perceptions surrounding these guidelines. Fifty-one variants were distributed to 20 participants from 10 institutions for classification using the new guidelines. Agreement was assessed using chance-corrected agreement (Cohen κ). κ was 0.35. To evaluate if data sharing could help resolve disagreements, a summary of variant classifications and additional information about each variant were distributed to all participants. κ improved to 0.7 after the original classifications were revised. Participants were invited to take a web-based survey regarding their perceptions of the guidelines. Only 20% (n = 3) of the survey respondents had prior experience with the guidelines in clinical practice. The main perceived barriers to guideline implementation included the complexity of the guidelines, discordance between clinical actionability and pathobiologic relevance, lack of familiarity with the new classifications, and uncertainty when applying criteria to potential germline variants. This study demonstrates noteworthy discordances between pathologists for variant classification in solid tumors when using the 2017 Association for Molecular Pathology/American Society of Clinical Oncology/College of American Pathologists guidelines. These findings highlight potential areas for clarification/refinement before mainstream clinical adoption. Laboratory testing in molecular oncology has changed considerably in the past decade. Multigene sequencing panels have gradually replaced single gene assays for the comprehensive genetic evaluation of solid tumors. The primary goal of solid tumor sequencing is to identify therapeutic, diagnostic, and prognostic biomarkers in the context of a specific tumor type to help guide clinical management decisions. However, translating the vast amount of data generated by next-generation sequencing into meaningful clinical information is a complex and often challenging process that requires a robust and reliable framework for curating, interpreting, and reporting actionable variants. First-generation reporting frameworks were primarily based on pathobiologic impact, but different frameworks adopted by individual laboratories had potential for poor interrater reliability. The Association for Molecular Pathology (AMP), the American Society of Clinical Oncology (ASCO), and the College of American Pathologists (CAP) jointly released guidelines in 2017 to address these issues of standardization and provide an evidence-based framework for the classification, interpretation, and reporting of sequence variants in cancer (Table 1).1Li M.M. Datto M. Duncavage E.J. Kulkarni S. Lindeman N.I. Roy S. Tsimberidou A.M. Vnencak-Jones C.L. Wolff D.J. Younes A. Nikiforova M.N. Standards and guidelines for the interpretation and reporting of sequence variants in cancer: a joint consensus recommendation of the Association for Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists.J Mol Diagn. 2017; 19: 4-23Abstract Full Text Full Text PDF PubMed Scopus (861) Google Scholar In contrast to previous classification schemes, the AMP/ASCO/CAP tier-based framework focuses on clinical significance and actionability. The intent of this shift was to highlight and prioritize detected alterations based on clinical evidence and streamline care decisions. However, actionability assessment of variants is challenging. Various pathologists may reach different conclusions regarding the classification of variants. Reliability is a fundamental property of any measurement or classification method. To our knowledge, the reliability of the 2017 AMP/ASCO/CAP guidelines has not been tested. The objective of this study was to evaluate the reliability of the guidelines. A secondary objective was to assess current practices and perceptions surrounding these guidelines.Table 1Summary Table of Variant ClassificationAdapted from Li et al.1Li M.M. Datto M. Duncavage E.J. Kulkarni S. Lindeman N.I. Roy S. Tsimberidou A.M. Vnencak-Jones C.L. Wolff D.J. Younes A. Nikiforova M.N. Standards and guidelines for the interpretation and reporting of sequence variants in cancer: a joint consensus recommendation of the Association for Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists.J Mol Diagn. 2017; 19: 4-23Abstract Full Text Full Text PDF PubMed Scopus (861) Google ScholarVariant categoryClinical impactTier IVariants of Strong Clinical Significance•Variants with FDA-approved therapy•Included in professional guidelines•Expert consensus regarding clinical significance (therapeutic, prognostic, or diagnostic) based on well-powered studiesTier IIVariants of Potential Clinical Significance•FDA-approved therapies for different tumor types•Investigational therapies•Limited clinical consensus regarding clinical significance (therapeutic, prognostic, or diagnostic) based on multiple small studies, preclinical trials, or few case reportsTier IIIVariants of Unknown Clinical Significance•Not observed at a significant allele frequency in population or cancer databases•No convincing published evidence of cancer associationTier IVBenign or Likely Benign Variants•Significant allele frequency in the population databases•No existing published evidence of cancer associationFDA, US Food and Drug Administration. Open table in a new tab FDA, US Food and Drug Administration. The study was performed according to the University of Utah (Salt Lake City, UT) Institutional Review Board–approved protocol for research use of deidentified data. The objective of this study was to determine whether classifications based on the 2017 AMP/ASCO/CAP guidelines are reliable. That is, using the guideline, are different users likely to achieve the same result? It was reasoned that experienced molecular pathologists and doctoral-level professionals working with clinical grade oncology next-generation sequencing tests at academic centers would be most likely to obtain concordant results. If this highly trained group had difficulty implementing the guidelines into practice, then it would be likely that others would also have trouble. The objective was not to estimate the overall reliability of molecular pathologists across the United States. Rather, the objective was to conduct a pilot study on a select group to probe for potential issues with reliability. Validity (accuracy) is also an important property of a measurement instrument. Raters may agree but may be incorrect. Therefore, both reliability and validity should be assessed. This study was designed as a first step toward that goal. It was reasoned that reliability should be assessed first because assessing validity has little value unless raters show reasonable reliability. Directors at 10 academic molecular diagnostics laboratories from different geographic areas of the United States were contacted. These laboratories routinely perform next-generation sequencing assays for solid tumors for clinical patient management decisions, are associated with major cancer centers, and are Clinical Laboratory Improvement Amendments certified. A participant is defined as an individual who performed variant classifications. Each laboratory could supply one or more participants. The number of participants from each laboratory ranged from 1 to 4 (mean, 2), for a total of 20 participants. The participants included molecular pathologists (n = 16), doctoral-level laboratory directors (n = 3), and a doctoral-level clinical variant scientist. All participants had formal training in molecular oncology and had approximately 1 to 7 years of experience reporting next-generation sequencing data. Each of the participants was asked to contribute five deidentified variants that had been detected by solid tumor sequencing in his/her clinical practice. The goal was to obtain a set of variants that were distributed across different genes, variant classes, and tumor types. Sixty-two deidentified variants were received. Two authors (L.V.F. and D.S.) reviewed these variants and selected 51 for in the variants were because were specific for variant was to that the selected variants were distributed across different variant and variant was not selected by a formal The included variants were distributed as five number and one Participants were also asked to information about the cancer gene and and associated with each of these variants. These data were to the The set was distributed to all participants. Participants were asked to the variants in the set using the 2017 AMP/ASCO/CAP guidelines and to the evidence that was to each A summary of classification and data by participants to their classifications for each of the 51 variants was and reviewed Table the and to PubMed Scopus Google A. A. A. as a for in PubMed Scopus Google A. Li M. by in PubMed Scopus Google M. M. M. Li A. to of 2017; PubMed Scopus Google S. A. A. A. S. and as a of to in 2017; PubMed Scopus Google A. M. M. with are to the of 2017; PubMed Scopus Google D.J. in cancer to Scopus Google A. A. evaluation and cancer assessment of PubMed Scopus Google et and the of and M. a new of in PubMed Scopus Google M. to 2017; PubMed Scopus Google C.L. A.M. M. The major of are and are associated with to the PubMed Scopus Google S. M. A case of in a 2017; Full Text Full Text PDF PubMed Scopus Google M. M. S. of in PubMed Scopus Google of in of tumor PubMed Scopus Google A.M. to tumor and to PubMed Scopus Google M. the cancer PubMed Scopus Google S. A. S. S. S. S. S. M. M. A. gene in cancer and on to PubMed Scopus Google Li Molecular of with PubMed Scopus Google The of the for and cancer Drug PubMed Scopus Google M. S. A. M. S. A. M. M. Clinical and of and PubMed Scopus Google S. M. gene in of the Full Text Full Text PDF PubMed Scopus Google M. S. M. of the gene in Mol Scopus Google M. A. A. D.J. M. of the gene in solid tumors and PubMed Scopus Google S. of by the PubMed Scopus Google M. C.L. for reporting of secondary findings in clinical and a of the American College of and 19: Full Text Full Text PDF PubMed Scopus Google Scholar This with deidentified was then to all participants. The participants were asked to their classifications in of this were asked whether would any of their variants. The variants were and to determine whether data sharing between different professionals improved classification between participants with the original classifications. This data sharing an for participants to their classifications. that sharing data would help resolve in variant and the overall variant classification agreement was according to published guidelines for agreement The of a for of reliability Full Text Full Text PDF PubMed Scopus Google S. A. A. M. for Reliability and Agreement were Full Text Full Text PDF PubMed Scopus Google Scholar data had = of are to case and case and M. D.J. using multiple case for a in when are PubMed Scopus Google on that are often to case often be when the number of is the number of in our study was our study between 20 each value had the potential to could in case and were in the Google Scholar were using the consensus classification were performed using College and agreement was assessed using the observed agreement and chance-corrected agreement Agreement was not κ was using the of agreement Scopus Google Scholar The in κ was using the of the of agreement for PubMed Scopus Google Scholar The for group were for using the multiple on multiple in clinical PubMed Scopus Google Scholar The number of the of and the for each were In also agreement for different of variants number and A for could not be because was one It was that variants with poor agreement the original classification would show the in was to whether the in agreement was to the original were in the to laboratories are using the new guidelines if not using whether laboratories to the guidelines in the were also in in using the new guidelines in that may reliability. To that participants were asked to to a web-based survey The survey is in the Table The overall observed interrater agreement was (Table and Table agreement for individual variants ranged from to between and and and The chance-corrected agreement was to The κ from case were the same as when were The κ for individual variants ranged from to to of Agreement and Variant are for multiple of type number to table the in agreement using the original and guidelines. The table agreement by institutions with more one and by variant not are for multiple of Open table in a new tab The table the in agreement using the original and guidelines. The table agreement by institutions with more one and by variant not Agreement was not on the type of agreement for variants = was the chance-corrected agreement for number = and variants = the was not improved agreement in of 51 agreement from to after participants had an to their original classifications (Table each participant changed 1 to of the original classifications Table and Table The in agreement was not associated with the original of agreement = in from the to for Each Open table in a new tab in agreement as a of original raters each variant according to the 2017 Association for Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists Standards and for the and of in were In the first raters were information to variants. In the raters were with the classifications of raters and the to their Each a The of this was to whether the in agreement on the original of The is the for the between in agreement original and the original The was not significant = = 20 to the Only 20% were using the new guidelines in their clinical practice. the that to the new guidelines a participants that the guidelines had the potential to reporting practices and the classification of variants Table who had not adopted the AMP/ASCO/CAP guidelines, that were using a classification and that a classification the one by the new guidelines. participants a for additional on classification of genetic as The main perceived barriers to implementation included the complexity of the guidelines, discordance between clinical actionability and pathobiologic relevance, lack of familiarity with the new classifications, and uncertainty when applying criteria to potential germline variants. This study interrater agreement for variant classification using the 2017 AMP/ASCO/CAP guidelines. The AMP/ASCO/CAP guidelines were generated to provide a and to variant classifications However, our findings significant interrater discordance for variant classifications using the new AMP/ASCO/CAP guidelines. The of interrater reliability is because the study was performed by a set of professionals at major academic cancer that this group would provide a for interrater reliability is important because classifications have clinical a in classification from clinical to of could for clinical or of these more variants not by oncology sequencing as in and the variant in for evidence is in the published However, classification between and were also in genes, as and all of the tumors that were included in the study Table The is because the clinical actionability of variants is that for In may be in with tumors that are type at because of the of that of in the of of PubMed Scopus Google of in PubMed Scopus Google Scholar Therefore, variants should not be significant unless has been in a clinical of the classifications clinical to the five variants with in in in a and in were potential germline variants in that have been into clinical as of cancer as of cancer or for is for the of PubMed Scopus Google Scholar Raters also had poor agreement with an of the gene in a for this variant were distributed across all on the classification and on the in and 4 and are of to with in A.M. D.J. Molecular biomarkers for the evaluation of cancer: guideline from the American Society for Clinical Pathology, College of American Association for Molecular Pathology, and the American Society of Clinical 2017; PubMed Scopus Google Scholar is data to that may in to in this tumor S. A. S. S. S. S. S. M. M. A. gene in cancer and on to PubMed Scopus Google Scholar The agreement in this variant may have been to uncertainty regarding the classification of an that in a significant gene Table A noteworthy classifications included a variant in as by participants and by participants. A of this variant would be for in prognostic in and in as in professional A.M. D.J. Molecular biomarkers for the evaluation of cancer: guideline from the American Society for Clinical Pathology, College of American Association for Molecular Pathology, and the American Society of Clinical 2017; PubMed Scopus Google Scholar that the classifications for this variant in cancer may have been to in evidence across different for of these as or may not management decisions, as these variants are to most and the of guideline is to across a reliable for classification of the and provide the most to management decisions. the actionability of a variant may the for variant by current the American College of and and guidelines for interpretation and classification of germline variants also observed in variant classification across different A pilot S. A. S. of guidelines laboratories in the Clinical Full Text Full Text PDF PubMed Scopus Google Scholar discordance in variant classifications across laboratories when applying the S. S. S. M. Laboratory and guidelines for the interpretation of sequence a joint consensus recommendation of the American College of and and the Association for Molecular Full Text Full Text PDF PubMed Scopus Google Scholar A study assessing interpretation for variants to an overall of S. S. Clinical laboratories to resolve in variant to 2017; 19: Full Text Full Text PDF PubMed Scopus Google Scholar In both studies, classification to and when data and classifications were the S. A. S. of guidelines laboratories in the Clinical Full Text Full Text PDF PubMed Scopus Google S. S. Clinical laboratories to resolve in variant to 2017; 19: Full Text Full Text PDF PubMed Scopus Google Scholar in this variant classification agreement across all improved by sharing variant classifications and evidence the participants in an to a consensus these findings highlight the of as a of discordances and of variant The of data sharing for of variant has also been by findings from S. S. A. and implementation of and sequencing for the of actionable number and gene in clinical cancer Mol Diagn. Full Text Full Text PDF PubMed Scopus Google S. A. S. S. Open sharing as a reporting on and the Open Full Text Full Text PDF PubMed Scopus Google interpretation of genetic variants and cancer by laboratories as assessed by the of PubMed Scopus Google Scholar and is by the of of and clinical data sharing is to genetic a of the American College of and 2017; 19: PubMed Scopus Google Scholar are potential for the classification discordances observed in this on evidence and lack of The AMP/ASCO/CAP classification focuses on evidence of actionability and on the pathobiologic of the variants in gene The on actionability the clinical of solid tumor sequencing because therapeutic, diagnostic, and prognostic information select the this is for that may contribute to but for actionability may be to clinical or case variants and to may the a laboratory may a In in of the variant classification by individual laboratories before the of these guidelines, the of variants have as a major for of variants as likely or variants of when classification are Therefore, it may be to to a variant of significance to a of potential clinical of variants on of Each of evidence be to a This process of evidence is challenging because evidence in and evidence for or information is for In the information is often and across multiple The guidelines not provide specific and for the and of evidence to be for variant for different evidence and are not by the guidelines. classification be when evidence is one of the 10 evidence by this for M.M. Datto M. Duncavage E.J. Kulkarni S. Lindeman N.I. Roy S. Tsimberidou A.M. Vnencak-Jones C.L. Wolff D.J. Younes A. Nikiforova M.N. Standards and guidelines for the interpretation and reporting of sequence variants in cancer: a joint consensus recommendation of the Association for Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists.J Mol Diagn. 2017; 19: 4-23Abstract Full Text Full Text PDF PubMed Scopus (861) Google Scholar is based on actionability Food and Drug professional guideline and of variants that lack evidence in this is likely to be more because it on of evidence that are and not or in any it is data and the of a variant in in to each in of when different of evidence are for a variant the guidelines not provide specific for of different of clinical therapeutic, diagnostic, or in different cancer types. in variant often because of in to curating, and and because of in the and of Therefore, the guidelines should the each of evidence is and This would of use and reliability of the The of a based on evidence to the one by the guidelines of germline variant classification, be S. S. S. M. Laboratory and guidelines for the interpretation of sequence a joint consensus recommendation of the American College of and and the Association for Molecular Full Text Full Text PDF PubMed Scopus Google S. S. Clinical laboratories to resolve in variant to 2017; 19: Full Text Full Text PDF PubMed Scopus Google S. S. A. and implementation of and sequencing for the of actionable number and gene in clinical cancer Mol Diagn. Full Text Full Text PDF PubMed Scopus Google S. A. S. S. Open sharing as a reporting on and the Open Full Text Full Text PDF PubMed Scopus Google interpretation of genetic variants and cancer by laboratories as assessed by the of PubMed Scopus Google of and clinical data sharing is to genetic a of the American College of and 2017; 19: PubMed Scopus Google M. C.L. the clinical validity of an evidence-based framework by the clinical 2017; Full Text Full Text PDF PubMed Scopus Google Scholar a would additional and this of variant classification across different laboratories and A as the Clinical S. A. S. S. A. a for assessing of genetic 2017; PubMed Scopus Google Scholar could be to the of guideline an streamline the of the guidelines, data and the potential for and the of in a A would also to the classification and for the of guidelines, as the for the evaluation of cancer of experience with the guidelines may have also to the agreement that was observed in this to our participants were using the 2017 AMP/ASCO/CAP guidelines in their of the 20 participants had experience with the guidelines. and as Variant both before and after the of the guidelines, are that users to and experience with the before clinical adoption. In and training in the guideline process is to In this of the survey respondents lack of familiarity with the new AMP/ASCO/CAP variant classification and reporting as one of the major to the implementation of the guidelines at their Therefore, a to the implementation and of variant classification guidelines should be molecular tumor could also as to and care on the and of the guidelines for evaluation of the clinical significance of variants and to at management decisions. The and classification of germline variants that may be tumor sequencing is also not by the AMP/ASCO/CAP guidelines. The guidelines the use of the criteria for these S. S. S. M. Laboratory and guidelines for the interpretation of sequence a joint consensus recommendation of the American College of and and the Association for Molecular Full Text Full Text PDF PubMed Scopus Google C.L. American College of for reporting of findings in clinical and Full Text Full Text PDF PubMed Scopus Google Scholar but more specific on and criteria for evidence and germline alterations was by participants of this study as a for specific of the germline variant classification guidelines into the classification could be in of the guidelines. This study had In this study was not to estimate an agreement that would across the reasoned that agreement in a group of would likely in agreement To that the study was based on a of 20 participants from 10 different laboratories to be of were criteria for across the may be more concordant in their variant when applying the new guidelines. The was small but for this This study is by the that most of the participants not have experience using the classification by the 2017 AMP/ASCO/CAP guidelines in their the participants were selected for their in molecular may have not had in the of the The was that in molecular diagnostics would be in applying the guidelines. study is to determine the amount of experience for pathologists to the guidelines. This study is also because was not into this study would have been if had been is for classification of variants because of the in the that agreement was as a for that it is that a group of could all be but that classifications with agreement are most likely the that the is in with of If agreement had been would not whether was also However, in this agreement was that was that agreement is with The AMP/ASCO/CAP guidelines a significant step toward of variant classification across laboratories but could from a with to address of the by this as these guidelines more and the complexity of and it that an between different be with any However, data from laboratories in a variant as may for In of the guidelines, should also be to the potential for and specific of the AMP/ASCO/CAP variant classification framework for interpretation of variants in as for germline A. S. A. M. and of the variant classification framework for by Full Text Full Text PDF PubMed Scopus Google A.M. A. M. A. Clinical of the variant interpretation guidelines for genetic PubMed Scopus Google Scholar of the guidelines laboratories to of their current reporting to the AMP/ASCO/CAP guidelines. may the existing and with individual laboratory information and variant classifications different for These are all that be In this study that in molecular diagnostics poor interrater reliability in the of the AMP/ASCO/CAP guidelines. Reliability should be an of the of a classification for in of this with Table with Table with Table