A Medical Image Challenge Infrastructure
K Farahani1*, J Kalpathy-Cramer2, B Bearce3, U Wagner4, (1) National Cancer Institute, Bethesda, MD, (2) Boston, MA, (3) No institution provided (4) Frederick National Lab, Frederick, MD
Presentations
PO-GePV-M-248 (Sunday, 7/25/2021) [Eastern Time (GMT-4)]
Purpose: To develop an infrastructure for benchmarking the performance of task-specific software tools in medical imaging challenge competitions.
Methods: The open-source challenge platform, CodaLab, has become a popular tool for benchmarking scientific competitions. We customized an instance of CodaLab for use in medical imaging applications, by adding capabilities to accept challenge proposals, provide guidelines, and distribute data from public image repositories such as The Cancer Imaging Archive, or from a private server. Additional developments include allowing docker submissions, a documentation site, and a helpdesk to assist users.
Results: The National Cancer Institute’s Medical Imaging Challenge Infrastructure (MedICI - https://github.com/QTIM-Lab/MedICI) can be deployed locally or in a public cloud. It allows for challenge design submissions. Organizers are able to use scripts and codes from public libraries in R, Python, Javascript, etc to evaluate the results based on previously defined metrics. To date, over 20 medical imaging challenges, with applications in cancer have been conducted through MedICI, either independently or in conjunction with scientific conferences. These have included challenges in benchmarking tools for image-based detection, diagnosis, classification of cancers, and prediction of response to radiotherapy. Many of the tools developed in response to challenges on MedICI have deployed innovative approaches to machine learning and deep learning in medical imaging.
Conclusion: Challenges provide a robust way to promote open data science by assessing the performance of software tools in addressing practical technical or clinical tasks in medical imaging. They facilitate a head-to-head assessment of innovative tools for diagnosis and treatment of disease using standardized datasets and high-quality ground truth data. As a resource, MedICI provides a comprehensive platform for the implementation of challenges in medical imaging. It will continue to evolve and be a valuable resource to the community by serving challenges to scientific organizations, including the AAPM.
ePosters
Keywords
Image Analysis, Image Processing
Taxonomy
IM/TH- Informatics: Informatics in Imaging (general)
Contact Email