Hazard Testing to Probe Points of Weakness in Automated Planning Tools
K Nealon1,2*, R Douglas2, E Han1,2, S Kry1,2, V Reed2, S Simiele1,2, L Zhang2, L Court1,2 (1) University of Texas MD Anderson UTHealth Graduate School of Biomedical Sciences, (2) University of Texas MD Anderson Cancer Center, Houston, TX
Presentations
MO-C930-IePD-F4-3 (Monday, 7/11/2022) 9:30 AM - 10:00 AM [Eastern Time (GMT-4)]
Exhibit Hall | Forum 4
Purpose: Hazard scenarios were created to assess and reduce the risk of planning errors in automated planning processes. This was accomplished through iterative testing and improvement of examined user interfaces.
Methods: Automated planning requires three user inputs: a CT, a planning directive, and contours. We investigated the ability of users to catch errors that were intentionally introduced into each of these three stages, according to an FMEA analysis. Reviewers underwent video training prior to reviewing and providing feedback for various test plans. Five radiation therapists each reviewed 15 patient CTs, containing 3 errors: inappropriate field of view, incorrect superior border, and incorrect identification of isocenter. Five radiation oncologists reviewed 10 prescription documents, containing 2 errors: incorrect prescription and treatment site. Five physicists reviewed 10 contour sets, containing 2 errors: missing contour slices and inaccurate target contour.
Results: Initially, 75% of hazards were detected in the service request approval. The visual display of prescription information was then updated to improve the detectability of errors based on user feedback. The change was then validated with five new oncologists who detected 100% of errors present. 83% of hazards were detected in the CT approval portion of the workflow. For the contour approval portion of the workflow, only 12.5% of hazards were detected by physicists, indicating this step should not be used for quality assurance. To mitigate the risk from errors at this step, users must perform a thorough review of contour quality prior to final plan approval.
Conclusion: Hazard testing was used to pinpoint the weaknesses of an automated planning tool and as a result, subsequent improvements were made. This study identified that not all workflow steps should be used for quality assurance and demonstrated the importance of performing hazard testing to identify points of risk in automated planning tools.
Funding Support, Disclosures, and Conflict of Interest: This work was partially funded by Varian Medical Systems
Keywords
Taxonomy
Contact Email