Quantitative dataset audit methods to identify bias, shortcut, and robustness risks for data-driven healthcare AI

About This Grant

Project Summary Machine learning and artificial intelligence (ML/AI) promise to enhance the quality, safety, efficiency, efficacy, affordability, and accessibility of healthcare. The anticipated benefits are evident from the surge in FDA ap- provals for AI/ML-based “software as a medical device (SaMD)”, increasing to over 600 in 2023. Approval requires adequate algorithm performance under controlled test conditions, but AI/ML solutions often deteriorate significantly in real-world settings, with controlled tests overestimating ML/AI performance by 20% or more. Thus, real-world deployment of ML/AI may yield substantially lower benefits or even net losses. The brittleness of ML/AI models is fundamentally linked to the data they are trained and tested on. Dataset bias can manifest in many ways, leading to unfair outcomes, poor generalization, and unpredictable behavior of AI/ML models. Identify- ing and quantifying dataset bias is thus most significant to prevent erroneous analyses, statistical fallacies and representational errors when using these datasets to develop and test AI/ML solutions. This proposal aims to develop automated techniques to identify dataset biases likely to affect down- stream ML/AI models. Using testbeds in medical image analysis, we will identify bias in the form of dataset shortcuts – non-clinically relevant features leaking task-relevant information, and imaging conditions – image properties associated with a specific clinical setting. In contrast to prior work that has focused on trained ML/AI models, our methods will audit datasets explicitly and independently of any assumptions about the ML/AI models that may use these datasets. This approach will help AI/ML SaMD stakeholders quantify and measure how well ML/AI methods address undesirable biases. Our specific aims are: 1) A framework and testbeds for assessing and validating dataset audits: We will create a framework to manipulate dataset bias, shortcuts, and imaging conditions for testing our auditing methods. We will curate datasets for chest X-ray, CT, and dermascopy, including public and in-house data, and define tasks for each modality, demonstrating the generality and utility of dataset auditing across varied medical imaging modalities. 2) Causality-driven auditing methods to identify and measure medical imaging dataset bias and shortcut learning risk: We will develop techniques to identify image-based signatures linked to data factors spuriously associated with outcomes, i. e., shortcuts, that lead to poor ML/AI performance. 3) Causality-driven auditing method to assess the robustness risk: We will assess how dataset composition with respect to image quality affects training and testing of AI/ML models, and using causal models of image formation, will identify image generation factors that limit model robustness. While our developments focus on ML/AI-based medical image analysis due to its rapid clinical translation, the concepts are broadly applicable to other ML/AI-based digital health tools, contributing to the development of trustworthy AI/ML digital health tools that transform healthcare delivery.