Curbing the Errors From Automated Tools Based On An FMEA of An AI-Based Treatment Planning System
K Nealon1,2*, P Balter1,2, R Douglas2, D Fullen2, S Hernandez1,2, P Nitsch2, A Olanrewaju2, M Soliman2, L Court1,2, (1) UT MD Anderson UTHealth Graudate School of Biomedical Sciences, Houston, TX (2) The University Of Texas MD Anderson Cancer Center, Houston, TX,
Presentations
SU-A-TRACK 6-1 (Sunday, 7/25/2021) 10:30 AM - 11:30 AM [Eastern Time (GMT-4)]
Purpose: To prospectively assess the patient risk associated with the clinical use of a web-based, automated contouring and treatment planning tool.
Methods: A Failure Modes and Effects Analysis (FMEA) was performed to quantify risk prior to clinical deployment of the Radiation Planning Assistant (RPA), an automated contouring and treatment planning tool. Potential errors were identified by a multidisciplinary team, consisting of representatives from physics, radiation therapy, dosimetry, physician, RPA development and quality improvement teams. Each failure mode was individually discussed and scored by all participants, over the course of twenty 90-minute meetings. A 1-10 scale for severity, occurrence and detectability was used, following TG-100 recommendations.
Results: Of the identified failure modes, 56.6% (164/290) were common in manual radiotherapy processes, leaving 126 unique to RPA assisted workflows. The mean RPN of these errors was 56.3, with a maximum of 486. The top ten failure modes were caused by automation bias, a tendency of users to rely too heavily on results from automated systems, operator and software errors. An action threshold of 125 was set, and corrective actions were taken to the 21 failure modes above that limit. As a result, changes were made to the RPA including modifications to the user interface to remove unnecessary input and better training to highlight the importance of review of the automated output. After these changes, the FMEA was rescored and the mean and maximum RPN decreased to 33.7 and 288, respectively.
Conclusion: The majority of high-risk failure modes were caused by automation bias or operator error, emphasizing the need to limit unnecessary user input while maximizing user review of output. As a result of this analysis, changes were made to the RPA software and RPN scores subsequently decreased, showing that the FMEA approach is an effective way to reduce risk for automated planning tools.
Funding Support, Disclosures, and Conflict of Interest: Partial funding for this work was provided by Varian Medical Systems
Handouts
Keywords
Taxonomy
Not Applicable / None Entered.
Contact Email