Using Multi-Agent System Powered by Large Language Model with Expert-in-the loop to Optimize Clinical Decision Support (MAPLE-CDS)

About This Grant

PROJECT SUMMARY The widespread adoption of electronic health records (EHRs), supported by over $34 billion in government in- vestment, has significantly increased the use of clinical decision support (CDS) systems. CDS provides up-to- date information and recommendations to healthcare professionals and patients to reduce errors and improve healthcare quality. However, CDS effectiveness is often hindered by a low acceptance rate, which is typically below 10%. High rates of low-relevance alerts lead to alert fatigue, desensitizing clinicians to alerts of higher importance. At Vanderbilt University Medical Center (VUMC), over 1,000 CDS alerts generate 70,000 user com- ments annually, and manual reviews of alerts based on medical literature are time-consuming and prone to delays, creating an urgent need for an automated system to generate suggestions to optimize CDS alerts. Large language models (LLMs) and multi-agent systems are promising tools to address this need. LLMs achieve high efficiency in processing large volumes of text, while multi-agent systems can collaborate to solve complex problems from multiple perspectives. In addition, we will incorporate a CDS-focused medical knowledge graph into the system to better retrieve relevant content, manage complex relationships in clinical data, and provide metadata (e.g., evidence strength). The overall objective of this proposal is to develop and evaluate an LLM- powered multi-agent system that integrates alert content, user feedback, and external knowledge sources to generate suggestions to improve CDS. Our central hypothesis is that system-generated suggestions outperform suggestions generated in the current manual processes. Our work includes three specific aims: Aim 1) Create a CDS-focused medical knowledge base and develop a graph for external knowledge support using LLMs, Aim 2) develop and validate a multi-agent system with LLM guardrails to generate CDS optimization suggestions, and Aim 3) evaluate and refine the multi-agent system via expert-in-the-loop feedback. The expected outcomes of this work include a scalable knowledge graph that integrates the latest medical knowledge, ensuring that CDS tools remain current and evidence-based. Additionally, the creation of an innovative LLM-powered multi-agent CDS audit system will improve the accuracy, relevance, and efficiency of CDS alerts, significantly reducing the manual effort required to maintain and update CDS content. The modularized architecture of the system could facilitate continuous improvement with new AI technology and expansion to other EHR-related tasks. While this project will focus on auditing current CDS alerts, the system’s potential applications include 1) helping CDS experts develop new CDS tools and 2) monitoring medical literature for updates that impact existing CDS tools and notifying stakeholders of relevant changes. Ultimately, this project aims to contribute to more intelligent and efficient CDS, improving patient safety and healthcare quality.