An Artificial Intelligence Approach to Understanding Trade-offs in Emergency Department Decision-making

About This Grant

Project Summary Emergency Department (ED) decision-making is inherently complex, influenced by uncertainty, time pressure, and high-stakes consequences. ED crowding exacerbates these challenges by increasing distractions and cognitive load, often leading to suboptimal decisions and adverse outcomes. Triage nurses are typically the first point of contact for medical needs assessment. For example, their role in prioritizing patients is one of the most consequential decisions, yet current triage accuracy is only about 60% compared to expert benchmarks. This performance gap stems from the complexity of ED scenarios, which often exceed human cognitive limits. Artificial Intelligence (AI), including Large Language Models (LLMs), offer the potential to support triage nurses and all members of the ED team by improving decision-making on many ED tasks, especially during crowding. Advantages of such tools include rapid processing of large volumes of data, operating without fatigue, and being deployable on demand – all while rivaling human decision-making on a wide variety of tasks. With recent advances in AI, there is a timely opportunity to investigate its utility in supporting ED decision-making. While AI/LLMs could be vital support for ED decision-making, there is no gold standard (expert annotated labeled data) by which to evaluate the quality or accuracy ED decisions, and no standardized tasks or metrics. We propose to develop a set of gold-standard ED decisions derived by expert annotation on patient cases for a set of decision-making tasks, and establish a set of metrics for assessing decision-making performance. In addition, we will fine-tune and validate fine-tuned LLMs, with the best-performing model termed “AI-Triage+”. We will then test the performance of AI-Triage+ on ED decision-making-tasks vs. status quo (human decisions) and existing pre-trained (generalist) AI models. AIM 1: Compare the performance of AI models to a gold standard on five ED decision-making tasks: ESI (Emergency Severity Index) level recommendation, Patient-facing diagnostic question generation, Acceptable safe wait time values for each patient, Recommend diagnostic tests and ED procedures for the patient, and Recommend ED resource level based on the patient's needs (aka patient disposition). We will assess overall model performance and performance on specific subsets of cases, e.g., by diagnoses, age group, ED workload (overcrowding score). AIM 2: Estimate the impact of AI models' decisions on clinical outcomes. Using existing risk equations, we will estimate how different ED decisions by AI models will impact ICU admissions or death in ED, hospital admissions, and length of hospital stay. We will also explore the potential of AI-Triage+ to improve ED efficiency at a system level. This work directly supports the NINR focus on systems and models of care, offering foundational tools and insights to guide AI integration into ED practice.