Purpose: To develop an infrastructure for benchmarking the performance of task-specific software tools in medical imaging challenge competitions.
Methods: The open-source challenge platform, CodaLab, has become a popular tool for benchmarking scientific competitions. We customized an instance of CodaLab for use in medical imaging applications, by adding capabilities to accept challenge proposals, provide guidelines, and distribute data from public image repositories such as The Cancer Imaging Archive, or from a private server. Additional developments include allowing docker submissions, a documentation site, and a helpdesk to assist users.
Conclusion: Challenges provide a robust way to promote open data science by assessing the performance of software tools in addressing practical technical or clinical tasks in medical imaging. They facilitate a head-to-head assessment of innovative tools for diagnosis and treatment of disease using standardized datasets and high-quality ground truth data. As a resource, MedICI provides a comprehensive platform for the implementation of challenges in medical imaging. It will continue to evolve and be a valuable resource to the community by serving challenges to scientific organizations, including the AAPM.