CAREER: Towards Gaze-guided Medical Image Analysis

About This Grant

Medical image interpretation is a complex, context-dependent process that requires nuanced reasoning and decision-making, honed through years of clinical experience. Eye gaze patterns, a rich source of implicit expert knowledge, reveal how clinicians systematically identify and interpret critical features in medical images. This project seeks to integrate these expert visual search patterns into machine learning (ML) frameworks, addressing the current gap in how ML models interpret medical images. By leveraging expert eye gaze as a privileged data source, the project aims to enhance diagnostic accuracy, improve model trustworthiness, and provide more interpretable and clinically relevant outputs. For example, understanding how a radiologist systematically examines a chest x-ray can help in designing algorithms that mimic this process, potentially improving the identification of subtle disease indicators that might be missed by less experienced readers or automated systems. By incorporating expert search patterns, ML models can become more accurate and reliable. The anticipated outcomes include improved clinical decision-making, augmented training for novice clinicians, and explainable artificial intelligence (AI) tools capable of advancing medical care and patient outcomes. This project will develop foundational methods to characterize and integrate dynamic gaze patterns into machine learning pipelines. Algorithms will be designed to capture the spatiotemporal nature of clinical experts’ visual search behaviors and their relationship to the spatial complexity of disease patterns in medical images. These gaze-informed models will leverage representations of scan paths to enhance both predictive performance and interpretability. Furthermore, gaze cues will be incorporated into multimodal models, enabling improved spatial and temporal coherence in tasks requiring image-text alignment, such as automated report generation. Evaluation of these methods will focus on challenging applications in radiology and histopathology, including cardiothoracic disease diagnosis, prostate cancer grading, and addressing class imbalance in long-tailed medical datasets. The project’s technical innovations aim to improve the predictive power of state-of-the-art deep learning models while introducing clinician-informed guidance, ensuring clinically trustworthy outputs. By bridging human expertise with computational models, this research addresses critical challenges in medical imaging and lays the groundwork for significant advancements in artificial intelligence-enabled healthcare solutions. The educational component of this project will promote interdisciplinary collaboration, fostering the next generation of researchers and clinicians adept at developing and utilizing AI-driven healthcare solutions. The methods to be developed are broadly applicable and can potentially influence data analysis across other scientific domains where an observer's gaze plays a critical role in reasoning and decision-making tasks. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.