Automated Plan Checking Software Demonstrates Continuous and Sustained Improvements in Safety and Quality: A 3-Year Longitudinal Analysis
S Liu*, D Stuhr, Y Zhou, H Pham, J Xiong, J Mechalakos, S Berry, Memorial Sloan Kettering Cancer Center, New York, NY
Presentations
PO-GePV-T-142 (Sunday, 7/25/2021) [Eastern Time (GMT-4)]
Purpose: Perform a longitudinal analysis of the performance of our automated plan check tool (PCT) by retrospectively evaluating the number of errors identified in plans delivered to patients in three, month-long, data collection periods taken between 2017 and 2020.
Methods: 11 automated checkers were retrospectively run on 1169 external beam radiotherapy treatment plans identified as meeting the following criteria: planning target volume (PTV) based multi-field photon plans receiving a status of treatment approved in either March 2017, March 2018, or March 2020. The number of “passes” and “flags” were recorded. “Flags” were sub-categorized into “false flags”, “false flags due to naming conventions”, or “real flags”. 2x2 contingency tables using a two-tailed Fischer’s exact test were utilized to determine whether there were nonrandom associations between the output of PCT and whether the check was manual or automated at the original time of treatment approval.
Results: A statistically significant decrease in flags between the pre- and post-automation datasets was observed for four contour-based checkers, namely adjacent structures overlap, empty structures and missing slices, overlap between body and couch, and laterality, as well as a checker that determined whether the plan’s global maximum dose was within the planning target volume. Analysis of the origin of false flags resulted in recommendations for algorithmic or configuration modifications to circumvent such flags in the future.
Conclusion: Periodic and longitudinal review of the performance of automated software is essential for monitoring and understanding its impact on error rates as well as for optimization of the tool to adapt to regular changes of clinical practice. PCT has demonstrated continuous contributions to the safe and effective delivery of external beam radiotherapy to our patient population, an impact that extends beyond its initial implementation and deployment.
Funding Support, Disclosures, and Conflict of Interest: The plan check tool project is part of a co-development agreement between MSKCC, University of Michigan, and Varian Medical Systems. At the time of data collection Sean Berry held a grant from Varian Medical Systems unrelated to this work.
Keywords
Taxonomy
Contact Email