CAREER: Explainable AI for Real-Time Video Anomaly Anticipation

About This Grant

Video surveillance systems, which are now common in cities and workplaces, present a valuable opportunity to predict and prevent harmful incidents before they occur. This project seeks to create advanced technologies capable of analyzing video streams in real time to anticipate potential anomalies with high accuracy. When such events are predicted, the system can initiate preventive measures, such as alerting users or triggering automated interventions—like activating a car's brakes or stopping an escalator. Importantly, the system will also provide detailed explanations, enabling users to understand why their actions or environment were adjusted. By leveraging cutting-edge techniques to identify patterns that precede abnormal events and generate visual or text-based explanations, this project has the potential to transform public safety and operational efficiency. For example, in intelligent traffic systems, it could help reduce pedestrian accidents and fatalities. In warehouses, it could prevent losses caused by equipment mishaps. In air traffic control, it could serve as a critical safeguard to ensure safe and orderly flight operations. By enhancing safety, reducing risk, and supporting human decision-making, this work addresses a critical need and aligns with national priorities to promote prosperity and welfare. While significant advancements have been made in video processing through artificial intelligence (AI), the task of anticipating video anomalies remains largely underexplored. Addressing this challenge involves overcoming three critical barriers: the absence of robust, annotated real-world datasets for anomaly anticipation, the lack of efficient real-time algorithms for accurate anomaly prediction, and the difficulty in explaining the reasoning behind AI-based decision-making in anomaly anticipation scenarios. This project will address each of these challenges. First, the research team will create a framework, based on the concept of digital twins, to semi-automatically capture and annotate anomaly anticipation datasets. This framework will focus on scenarios with static camera positions, enabling the dynamic extraction and recomposition of real-world objects and their motion patterns to create novel motion anomalies. Generative AI models conditioned on the designed anomalous patterns will be used to create realistic videos containing anomalies. Second, the team will design real-time AI models capable of accurately predicting motion-based anomalies prior to their occurrence by leveraging multimodal models trained to anticipate the trajectories and positions of objects as well as the probability of an upcoming anomaly. Third, the project will develop descriptive visual AI models that can provide stakeholders with clear, understandable explanations of anticipated anomalies. As a case study, the proposed methods will be applied to traffic anomaly anticipation, demonstrating the practical utility of the research. The project will also integrate research and education by incorporating in-class assignments and organizing annual student competitions. The success of this project will lead to a new paradigm for training explainable anomaly detection models and expand the conversation from anomaly detection to real-time anomaly anticipation and prevention. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.