Advancing Intelligent Cognitive Load Sensing and Adaptive Scaffolding to Support Collaborative Simulation-based Learning in High-Stakes Environments

About This Grant

When clinicians receive high-quality team training for managing healthcare emergencies, such as in-hospital cardiac arrests, patients have a better chance of surviving. However, the high cognitive demands involved in complex decision making and team management can harm performance, particularly among healthcare professionals in training or in new roles. This project aims to understand and improve how medical professionals learn to work as an effective team by detecting and managing the mental demands they face during high-stakes events. By leveraging multimodal data (e.g., heart rate, speech, gaze) within team-based immersive virtual reality, this project enables trainee teams to practice in a controlled, simulated environment while receiving "just enough, just in time, and just for you" feedback at both individual and team levels. The ultimate goal is to equip trainees with strategies for making rapid, accurate, and repeatable decisions while effectively executing tasks to save lives. The project's outputs, including an open-source database documenting types of cognitive load triggers and corresponding strategies for regulating cognitive load, are designed to support a wide range of stakeholders, including medical educators, quality and safety professionals, human factors engineers, and those developing cardiac arrest response guidelines. The training methods developed in this research could also benefit other fields that rely on expert teams, including aviation, emergency rescue operations in the military, and wildfire management, leading to safer and more effective teamwork in high-stakes situations. To meet these goals, this project integrates multimodal sensing, modeling, and instructional strategies to support regulation of cognitive load at both individual and team levels during collaborative learning tasks. Unlike prior work, which relied on noisy single modalities and self-report measures after performance events, this integrated approach provides a comprehensive framework for detecting, modeling, and responding to cognitive load in a complex VR simulation-based training environment. In its first phase, the project will model cognitive load using multimodal signals such as visual, linguistic, and physiological responses, including interactions between team members. The second phase will involve qualitative interviews with learners to elucidate their cognitive overload experiences that correspond to the cognitive load peaks and behavior patterns identified in the first phase. These findings, along with the extracted multimodal features, will be used in phase three to detect and model cognitive load and develop AI-driven strategies. Finally, phase four will evaluate the impact of these findings on learners through a quasi-experimental study. Understanding markers that may predispose learners to errors or delays in therapeutic interventions will provide significant insight into a more holistic assessment of individual and team learning processes and provide unique opportunities for feedback, practice, and/or remediation. This project is funded by the Research on Innovative Technologies for Enhanced Learning (RITEL) program that supports early-stage exploratory research in emerging technologies for teaching and learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.