Purpose: To effectively address the scarce annotation problem, we proposed a Deep Reinforcement Learning (DRL) based pipeline approach for image segmentation. Unlike supervised machine learning, DRL is a self-learning algorithm to learn through the interactions with the given environment, and adjust the action based on observed state and received reward. We aim to develop a DRL-based approach for prostate gland segmentation on MRI by self-learning, and introduce a novel post processing approach based on the Kalman filter to enhance the segmentation accuracy in 3D volumes.
Methods: We considered the segmentation as a painting task and trained a DRL agent to draw the structure's boundary in a sequential decision-making approach. The pipeline has two phases: (a) a backbone encoder capturing the global information and providing the start point for the agent; (b) a DRL based agent to delineate boundary on 2D MRI by interacting with the environment. The segmented slices for each patient were then aggregated and post-processed using a Kalman filter to rectify the contour results in the 3D space. The proposed model was trained with SPIE-AAPM-NCI PROSTATEx-2 Challenge (SPIE) dataset, and evaluated on both SPIE and an in-house dataset. Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD) were used as the evaluation metrics.
Results: The proposed method achieved a DSC of 87.88%±3.0%, 87.89%±1.7% on the validation and test based on the 2D slices, and 85.20%±10.3% and 85.27%±10.4% on the 3D volumes using the SPIE data. The results were 84.18%±4.3% and 81.72%±11.7 on the 3D and 2D respectively using the in house data. The 3D 95% HD was 4.08±0.94 mm and 5.28±2.21 mm on the validation and test, 6.49±2.48 mm on the in-house dataset.
Conclusion: The DRL agent provides promising prediction results on organ segmentation and the Kalman filter-based post segmentation shows its effectiveness in alleviating the segmentation variance.
Funding Support, Disclosures, and Conflict of Interest: The work was supported by a Research Scholar Grant, RSG-15-137-01-CCE from the American Cancer Society.