Beyond the Usual Prediction Accuracy Metrics: Reporting Results for Clinical Decision Making
Editorials21 August 2012Beyond the Usual Prediction Accuracy Metrics: Reporting Results for Clinical Decision MakingA. Russell Localio, PhD and Steven Goodman, MD, MHS, PhDA. Russell Localio, PhDFrom University of Pennsylvania, Philadelphia, PA 19104, and Stanford University School of Medicine, Stanford, CA 94305. and Steven Goodman, MD, MHS, PhDFrom University of Pennsylvania, Philadelphia, PA 19104, and Stanford University School of Medicine, Stanford, CA 94305.Author, Article, and Disclosure Informationhttps://doi.org/10.7326/0003-4819-157-4-201208210-00014 SectionsAboutFull TextPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissions ShareFacebookTwitterLinkedInRedditEmail How well an algorithm or a test predicts or diagnoses disease is not an adequate determination of its clinical usefulness. The relevant question is, are people better or worse off if the test is used as part of clinical care? Stock measures of prediction accuracy, such as sensitivity, specificity, area under the receiver-operating characteristic (ROC) curve, Brier scores, or predicted-versus-observed tests for model calibration, do not measure patient outcomes after a test. Derived measures, such as likelihood ratios, or traditional association measures, such as odds ratios, also do not serve that purpose. Fortunately, a large and growing body of analytic ...References1. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128-38. [PMID: 20010215] CrossrefMedlineGoogle Scholar2. Raji OY, Duffy SW, Agbaje OF, Baker SG, Christiani DC, Cassidy A, et al. Predictive accuracy of the Liverpool Lung Project risk model for stratifying patients for computed tomography screening for lung cancer. A case-control and cohort validation study. Ann Intern Med. 2012;157:242-50. LinkGoogle Scholar3. Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980;302:1109-17. [PMID: 7366635] CrossrefMedlineGoogle Scholar4. Greenland S. The need for reorientation toward cost-effective prediction: comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M.J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med. 2008;27:199-206. [PMID: 17729377] MedlineGoogle Scholar5. Hunink MG, Glasziou PP, Siegel JE, Weeks JC, Pliskin JS, Elstein AS, et al. Decision Making in Health and Medicine: Integrating Evidence and Values. Cambridge, UK: Cambridge Univ Pr; 2001. Google Scholar6. Sox H, Marton KI, Blatt MA, Higgins MC. Medical Decision Making. Philadelphia: American Coll Physicians; 2006. Google Scholar7. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26:565-74. [PMID: 17099194] CrossrefMedlineGoogle Scholar8. Helfand M, Berg A, Flum D, Gabriel S, Normand SL, eds; PCORI Methodology Committee. Draft Methodology Report: Our Questions, Our Decisions: Standards for Patient-Centered Outcomes Research. Patient-Centered Outcomes Research Institute; 23 July 2012. Google Scholar9. Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P. Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol. 2004;159:882-90. [PMID: 15105181] CrossrefMedlineGoogle Scholar10. Software for Decision Curve Analysis. Memorial Sloan-Kettering Cancer Center; 2012. Accessed at www.mskcc.org/research/epidemiology-biostatistics/health-outcomes/collaborative/decision-curve-analysis on 16 July 2012. Google Scholar11. Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. 2008;8:53. [PMID: 19036144] CrossrefMedlineGoogle Scholar12. Baker SG. Putting risk prediction in perspective: relative utility curves. J Natl Cancer Inst. 2009;101:1538-42. [PMID: 19843888] CrossrefMedlineGoogle Scholar13. Pepe M, Longton G, Janes H. Estimation and comparison of receiver operating characteristic curves. Stata J. 2009;9:1. [PMID: 20161343] CrossrefMedlineGoogle Scholar Author, Article, and Disclosure InformationAffiliations: From University of Pennsylvania, Philadelphia, PA 19104, and Stanford University School of Medicine, Stanford, CA 94305.Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M12-1744.Corresponding Author: A. Russell Localio, PhD, Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, 635 Blockley Hall, 423 Guardian Drive, Philadelphia, PA 19104; e-mail, [email protected].Current Author Addresses: Dr. Localio: Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, 635 Blockley Hall, 423 Guardian Drive, Philadelphia, PA 19104.Dr. Goodman: Stanford University, 259 Campus Drive, HRP/Redwood Building, Stanford, CA 94305. PreviousarticleNextarticle Advertisement FiguresReferencesRelatedDetailsSee AlsoPredictive Accuracy of the Liverpool Lung Project Risk Model for Stratifying Patients for Computed Tomography Screening for Lung Cancer Olaide Y. Raji , Stephen W. Duffy , Olorunshola F. Agbaje , Stuart G. Baker , David C. Christiani , Adrian Cassidy , and John K. Field Metrics Cited byIntegrative nomogram of intratumoral, peritumoral, and lymph node radiomic features for prediction of lymph node metastasis in cT1N0M0 lung adenocarcinomasA novel risk-predicted nomogram for sepsis associated-acute kidney injury among critically ill patientsDevelopment and validation of a nomogram to predict in-hospital cardiac arrest among patients admitted with acute coronary syndromeDecision curve analysis to evaluate the clinical benefit of prediction modelsDevelopment a nomogram to predict fertilisation rate of infertile males with borderline semen by using semen parameters combined with AMH and INHBMinimum sample size for external validation of a clinical prediction model with a binary outcomeEstablishment of the Radiologic Tumor Invasion Index Based on Radiomics Splenic Features and Clinical Factors to Predict Serous Invasion of Gastric CancerRadiomics Nomogram for Identifying Sub-1 cm Benign and Malignant Thyroid LesionsA Nomogram for Predicting Risk of Thromboembolism in Gastric Cancer Patients Receiving ChemotherapyLeveraging Decision Curve Analysis to Improve Clinical Application of Surgical Risk CalculatorsReferencesDevelopment and Validation of a Nomogram Based on Motoric Cognitive Risk Syndrome for Cognitive ImpairmentDevelopment and validation of a simplified nomogram predicting individual critical illness of risk in COVID‐19: A retrospective studyA Predictive Nomogram for Red Blood Cell Transfusion in Pheochromocytoma Surgery: A Study on Improving the Preoperative Management of PheochromocytomaThe value of spleen-preserving lymphadenectomy in total gastrectomy for gastric and esophagogastric junctional adenocarcinomas: A long-term retrospective propensity score match study from a high-volume institution in ChinaNomogram based on preoperative CT imaging predicts the EGFR mutation status in lung adenocarcinomaCompeting endogenous RNA network identifies mRNA biomarkers for overall survival of lung adenocarcinoma: two novel on-line precision medicine predictive toolsCervical spine osteoradionecrosis or bone metastasis after radiotherapy for nasopharyngeal carcinoma? The MRI-based radiomics for characterizationPrediction of iatrogenic preterm birth in patients with scarred uterus: a retrospective cohort study in Northeast ChinaThe Value of Preseason Screening for Injury Prediction: The Development and Internal Validation of a Multivariable Prognostic Model to Predict Indirect Muscle Injury Risk in Elite Football (Soccer) PlayersDeep Learning Using Chest Radiographs to Identify High-Risk Smokers for Lung Cancer Screening Computed Tomography: Development and Validation of a Prediction ModelMichael T. Lu, MD, MPH, Vineet K. Raghu, PhD, Thomas Mayrhofer, PhD, Hugo J.W.L. Aerts, PhD, and Udo Hoffmann, MD, MPHDevelopment of a Novel Serum Exosomal MicroRNA Nomogram for the Preoperative Prediction of Lymph Node Metastasis in Esophageal Squamous Cell CarcinomaThe Unrealised Potential for Predicting Pregnancy Complications in Women with Gestational Diabetes: A Systematic Review and Critical Appraisal A PI-RADS-Based New Nomogram for Predicting Clinically Significant Prostate Cancer: A Cohort Study Nomogram for Predicting Risk of Digestive Carcinoma Among Patients with Type 2 Diabetes RETRACTED ARTICLE: The diagnostic nomogram of platelet-based score models for hepatic alveolar echinococcosis and atypical liver cancerIntegrated analysis reveals potential long non-coding RNA biomarkers and their potential biological functions for disease free survival in gastric cancer patientsPlasma sTNFR1 and IL8 for prognostic enrichment in sepsis trials: a prospective cohort studyA simple, step-by-step guide to interpreting decision curve analysisThe competitive endogenous RNA regulatory network reveals potential prognostic biomarkers for overall survival in hepatocellular carcinomaA nomogram based on a gene signature for predicting the prognosis of patients with head and neck squamous cell carcinomaExternal validation of a prediction tool to estimate the risk of human immunodeficiency virus infection amongst men who have sex with menA Systematic Review of the Literature Demonstrates Some Errors in the Use of Decision Curve Analysis but Generally Correct Interpretation of FindingsA nomogram for the prediction of cerebrovascular disease among patients with brain necrosis after radiotherapy for nasopharyngeal carcinomaReporting and Interpreting Decision Curve Analysis: A Guide for InvestigatorsPrognostic Significance of CSN2, CD8, and MMR Status-Associated Nomograms in Patients with Colorectal CancerPrognostic significance of nomograms integrating IL-37 expression, neutrophil level, and MMR status in patients with colorectal cancerRandom-effects meta-analysis of the clinical utility of tests and prediction modelsA prediction model for lymph node metastasis in T1 esophageal squamous cell carcinomaLes nomogrammes dans la pratique clinique : méthodologie, limites et intérêtsPrognostic significance of cystatin SN associated nomograms in patients with colorectal cancerNew Sepsis Definition (Sepsis-3) and Community-acquired Pneumonia Mortality. A Validation and Clinical Decision-Making StudyClinical Utility of Risk Models to Refer Patients with Adnexal Masses to Specialized Oncology Care: Multicenter External Validation Using Decision Curve AnalysisPrognostic and Predictive Value of p21-activated Kinase 6 Associated Support Vector Machine Classifier in Gastric Cancer Treated by 5-fluorouracil/Oxaliplatin ChemotherapyDevelopment and Validation of the PREMM 5 Model for Comprehensive Risk Assessment of Lynch SyndromePrognostic Value of E-cadherin-, CD44-, and MSH2-associated Nomograms in Patients With Stage II and III Colorectal CancerKey steps and common pitfalls in developing and validating risk modelsDevelopment and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal CancerThe indication area of a diagnostic test. Part I—discounting gain and loss in diagnostic certaintyGraphical assessment of incremental value of novel markers in prediction models: From statistical to decision analytical perspectivesPrognostic utility of serum CRP levels in combination with CURB-65 in patients with clinically suspected sepsis: a decision curve analysisCalibration of Risk Prediction ModelsTransparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and ElaborationKarel G.M. Moons, PhD, Douglas G. Altman, DSc, Johannes B. Reitsma, MD, PhD, John P.A. Ioannidis, MD, DSc, Petra Macaskill, PhD, Ewout W. Steyerberg, PhD, Andrew J. Vickers, PhD, David F. Ransohoff, MD, and Gary S. Collins, PhDTowards better clinical prediction models: seven steps for development and an ABCD for validationSensitivity and Specificity Can Change in Opposite Directions When New Predictive Markers Are Added to Risk ModelsNet Reclassification Improvement: Computation, Interpretation, and Controversies A Literature Review and Clinician's GuideMaarten J.G. Leening, MD, MSc, Moniek M. Vedder, MSc, Jacqueline C.M. Witteman, PhD, Michael J. Pencina, PhD, and Ewout W. Steyerberg, PhDDoes the Net Reclassification Improvement Help Us Evaluate Models and Markers?Andrew J. Vickers, PhD and Margaret Pepe, PhDPrediction of 1-Year Mortality in Patients With Acute Coronary Syndromes Undergoing Percutaneous Coronary InterventionComparative Effectiveness Research in OncologyIntroducing, OncoTarget 21 August 2012Volume 157, Issue 4Page: 294-295KeywordsCancer screeningClinical epidemiologyComputed axial tomographyDecision analysisDecision makingLung and intrathoracic tumorsLungsSpecificity ePublished: 21 August 2012 Issue Published: 21 August 2012 Copyright & PermissionsCopyright © 2012 by American College of Physicians. All Rights Reserved.PDF downloadLoading ...