The first steps in the evaluation of a "black-box" decision support tool: a protocol and feasibility study for the evaluation of Watson for Oncology
Journal Title: Journal of Clinical and Translational Research - Year 2018, Vol 3, Issue 3
Abstract
Background and aim: Medical specialists aim to provide evidence-based care based on the most recent scientific insights, but with the ongoing expansion of medical literature it seems unfeasible to remain updated. ”Black-box” decision support tools such as Watson for Oncology (Watson) are gaining attention as they offer a promising opportunity to conquer this challenging issue, but it is not known if the advice given is congruent with guidelines or clinically valid in other settings. We present a protocol for the content evaluation of black-box decision support tools and a feasibility study to test the content and usability of Watson using this protocol. Methods: The protocol consists of developing synthetic patient cases based on Dutch guidelines and expert opinion, entering the synthetic cases into Watson and Oncoguide, noting the response of each system and evaluating the result using a cross-tabulation scoring system resulting in a score range of −12 to +12. Treatment options that were not recommended according to the Dutch guideline were labeled with a ”red flag” if Watson recommended it, and an ”orange flag” if Watson suggested it for consideration. To test the feasibility of applying the protocol, we developed synthetic patient cases for the adjuvant treatment of stage I to stage III colon cancer based on relevant patient, clinical and tumor characteristics and followed our protocol. Additionally, for the feasibility study we also compared the recommendations from the NCCN guideline with Watson’s advice, and evaluated usability by a cognitive walkthrough method. Results: In total, we developed 190 synthetic patient cases (stage I: n=8; stage II: n=110; and stage III: n=72). Overall concordance scores per case for Watson versus Oncoguide ranged from a minimum score of -4 (n=6) to a maximum score of +12 (n=17) and from −4 (n=9) to +12 (n=24) for Watson versus the NCCN guidelines). In total, 69 cases (36%) were labeled with red flags, 96 cases (51%) with orange flags and 25 cases (13%) without flags. For the comparison of Watson with the NCCN guidelines, no red or orange flags were identified. Conclusions: We developed a research protocol for the evaluation of a black-box decision support tool, which proved useful and usable in testing the content and usability of Watson. Overall concordance scores ranged considerably between synthetic cases for both comparisons between Watson versus Oncoguide and Watson versus NCCN. Non-concordance is partially attributable to guideline differences between the United States and The Netherlands. This implies that further adjustments and localization are required before implementation of Watson outside the United States. Relevance for patients: This study describes the first steps of content evaluation of a decision support tool before implementation in daily oncological patient care. The ultimate goal of the incorporation of decision support tools in daily practice is to improve personalized medicine and quality of care.
Authors and Affiliations
Lotte Keikes, Stephanie Medlock, Daniël J. van de Berg , Shuxin Zhang, Onno R. Guicherit, Cornelis J. A. Punt,Martijn G. H. van Oijen
Keywords
Related Articles
- EP ID EP678756
- DOI -
- Views 230
- Downloads 0
Racial and ethnic disparities in surgical amputations following serious musculoskeletal infections in a diverse New Mexico cohort
Background: Patients with serious musculoskeletal infections may encounter health disparities across multiple phases of prevention and treatment, including surgical intervention. The purpose of this study was to identify...
Salivary nitrite content, cognition and power in Mixed Martial Arts fighters after rapid weight loss: a case study
Background and Aim: Rapid weight loss (RWL) is extensively practiced by combat sports athletes, including Mixed Martial Arts (MMA), but its effects on performance are not well established with different magnitudes of RWL...
Intravenous infusion of magnesium sulfate is not associated with cardiovascular, liver, kidney, and metabolic toxicity in adults
Background: Magnesium (Mg) deficiency contributes to the pathophysiology of numerous diseases. The therapeutic use of Mg has steadily increased over time. The increased in-hospital use of intravenous (IV) magnesium sulfa...
The effect of dietary supplementation on brain-derived neurotrophic factor and cognitive functioning in Alzheimer’s dementia
Background and Aim: The objective of the present study was to investigate the effect of an aloe polymannose multinutrient complex (APMC) on pro brain-derived neurotrophic factor (BDNF) and mature BDNF in persons with mod...
Neonatal circumcision and prematurity are associated with sudden infant death syndrome (SIDS)
Background: Sudden infant death syndrome (SIDS) is the most common cause of postneonatal unexplained infant death. The allostatic load hypothesis posits that SIDS is the result of cumulative perinatal painful, stressful,...