Task-Based Sampling of the MIDRC Sequestered Data Commons for Algorithm Performance Evaluation
N Baughan1*, H Whitney1,2, K Drukker1, B Sahiner3, T Hu3, G Kim4, M McNitt-Gray4, K Myers5, M Giger1, (1) University of Chicago, Chicago, IL, (2) Wheaton College, Wheaton, IL, (3) US Food and Drug Administration, Silver Spring, MD, (4) UCLA, Los Angeles, CA, (5) US Food and Drug Administration, retired, Silver Spring, MD
Presentations
MO-A-BRC-6 (Monday, 7/11/2022) 7:30 AM - 8:30 AM [Eastern Time (GMT-4)]
Ballroom C
Purpose: The Medical Imaging and Data Resource Center (MIDRC) is a multi-institutional effort to accelerate medical imaging machine intelligence research and create a publicly available image commons as well as a sequestered commons for performance evaluation and benchmarking of algorithms. The sequestered commons (approximately 20% of all images and metadata) will act as a large pool from which representative samples can be drawn to evaluate the performance of user-developed algorithms. To test a research claim on a specified population, a method must be developed to sample the sequestered commons that creates representative test sets while maintaining the integrity of the sequestered commons.
Methods: For each task, the goal is to create datasets used to test a research claim on a specified population. A method of optimized quota sampling was developed to draw from the sequestered commons a sampling frame that matches a specified population within a pre-determined margin. From the sampling frame, random samples of subjects are then selected to be used as the test set. For this study, two tasks were identified to illustrate the task-based sampling
methods: (1) binary classification as COVID+/COVID- and (2) ranking of COVID severity, with the specified population chosen to match the US Census.
Results: From an example database of 9504 subjects, sampling frames of 971 and 593 subjects were selected through the developed method of optimized quota sampling to match the specified population within a 1% margin. Then, from the sampling frames, two possible test sets of 100 and 60 subjects for each task, respectively, were randomly selected, resulting in approximately 10% overlap of subjects between the two test sets.
Conclusion: Samples produced from an example database demonstrated that the process of optimized quota sampling can draw multiple representative test sets from a large database while maintaining agreement with specified population characteristics.
Funding Support, Disclosures, and Conflict of Interest: This research was funded through The Medical Imaging Data Resource Center (MIDRC), which is funded by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health under contracts 75N92020D00021, 75N92020C0008, and AWD101462-A.
Keywords
Simulation, Optimization, Performance Tests
Taxonomy
IM- Dataset Analysis/Biomathematics: Machine learning
Contact Email