Challenges On Implementing An Hybrid AI-And-Rules Based Plan Check Tool in Clinical Practice - a Pilot Study
S Luk*, E Ford, M Kim, M Phillips, K Hendrickson, A Kalet, University of Washington, Seattle, WA
Presentations
PO-GePV-T-155 (Sunday, 7/25/2021) [Eastern Time (GMT-4)]
Purpose: Artificial Intelligence (AI) has become a focus of research in radiation oncology in recent years, demonstrating potential for improvements in clinical safety in multiple studies. However, there are hurdles to the implementation of AI applications in ways that can realize those improvements clinically. In this work, we report a pilot study on implementing an in-house developed hybrid rules-and-AI based tool designed to assist medical physicists in initial plan checks. The goals of the study are to identify the key challenges and evaluate these aspects of system implementation into clinical workflow.
Methods: The web-based plan check tool takes a hybrid method approach, consisting of a rules-based component and a Bayesian network (BN)-based AI model. In the pilot study, three volunteering clinical medical physicists use the tool to assist their routine plan check process. Potential errors that are flagged by the tool are verified by the physicists and report as true positive or false positive results. Feedback is collected from the volunteers on their concerns using the tool.
Results: The simple logic system has shown excellent performance in the pilot study, while the BN-based AI tool successfully flags different potential logical errors in clinical plans. However, the AI tool also reported multiple false positive flags, and volunteers have reported that these repeated false positive errors combined with the lack of understanding on the AI model reduced their confidence in relying on the tool.
Conclusion: The BN-based AI tool was found to be useful to physicists in identifying errors in treatment plans during routine checks. We also found that interpretability of results presented a challenge to users, and understanding how to manage trade-offs between reliability, efficiency, and efficacy of the model needs to be explored.
ePosters
Keywords
Quality Assurance, Bayesian Statistics
Taxonomy
IM/TH- Informatics: Informatics in Therapy (general)
Contact Email