Distributed foundational models for multi-task learning in diabetic retinopathy

About This Grant

Abstract: This project aims to establish distributed federated learning (FL) approaches for multi-task training of foundational machine learning (ML) models for diabetic retinopathy (DR), using multi-modal, real-world optical coherence tomography (OCT) data (OCT cross-section, OCT angiography (OCTA), and OCT enface). DR is one of the leading causes of severe vision loss. Early detection, prompt intervention, and reliable assessment of treatment outcomes are essential to prevent irreversible vision loss from DR. However, there are major challenges towards developing clinically relevant holistic algorithms that can perform multi-tasks, i.e., multi-class classification of disease stages (diagnosis), prediction of onset and progression of disease stages (prognosis), and assessment of treatment outcomes. They require large amounts of well curated and labelled datasets from a diverse sub-population for robust performance. Moreover, efforts towards large, centralized datasets for ML research are hindered by significant barriers to data sharing and privacy concerns. In this project, we propose to develop foundational ML models that allow efficient learning of feature representations from a large corpus of ophthalmic imaging data for various downstream tasks – breaking the task-specific paradigm of current ML models. We also establish novel federated ML approaches, where the model training is distributed across institutions instead of sharing patient data. Our first aim is to establish and validate a domain adaptive FL framework for DR diagnosis across four independent institutions. We propose a novel ophthalmic adaptive personalized FL (optho-APFL) technique to tackle domain shift caused by heterogeneous data distribution at different institutions (due to different sub-population density and OCT devices/imaging protocols). We will conduct experiments on the FL deployment in a clinical setting and integrate a granular differential privacy (DP) algorithm into our FL framework to provide ‘patient-level’ data privacy. Key success criterion is to deploy the FL framework and validate FL-trained ML models against state-of-the-art models for DR diagnosis. The second aim is to develop foundational ML models with self-supervised learning (SSL) to learn multiple tasks within the same framework from label invariant OCT/OCTA data, where different institutions don’t need to have labeled data for each of the tasks. We will train these foundational models in a centralized and FL framework for comparative analysis. Key success criterion is to i) validate foundational model performance for multi-task learning (MTL) (DR staging, prediction of NPDR to PDR progression, and prediction of DME treatment evaluation) on new clinical data (centralized and FL approach, and ii) identify task-specific quantitative OCT/OCTA (mean and artery-vein specific) features. As an alternative approach, we propose diffusion probabilistic modeling (DPM) for SSL to learn holistic representations from multi-modal OCT data for MTL, and to explore dynamic federated averaging approaches. Success of this project will establish OCT/OCTA based distributed foundational models for objective MTL in DR using label-invariant data across multi-institutions and standardize OCT/OCTA features for MTL.