SHF: SMALL: FIRST: Enabling Real-Time, Scalable, and Verifiable Generative Visual AI Systems

About This Grant

Generative Artificial Intelligence (AI) has revolutionized visual content creation, enabling the production of highly realistic, synthesized video at an unprecedented scale. However, adapting current image-based diffusion methods for real-time, long-form video editing faces significant challenges, including latency, maintaining consistency between frames, and hardware efficiency. These challenges restrict the wider application of generative AI systems in various fields such as research, education, healthcare. This project introduces a new AI framework designed for real-time, scalable, and precise generative video editing, with a strong focus on rigorous algorithms, efficient system-level operations, and verifiable results. This project will help advance national interests in economic competitiveness, education, and public welfare by making generative visual AI systems more efficient and broadly accessible to a variety of users. This project advances the state of the art in generative visual AI systems through four key innovations: motion-adaptive cross-frame attention, pipelined frame scheduling for multi- graphics processing unit (GPU) systems, formal verification of semantic consistency, and system-level validation on real hardware. Motion-adaptive attention tracks scene changes, trimming computation while preserving spatio-temporal coherence. A pipelined, concurrency-optimized scheduler distributes encoding and decoding across multiple GPUs for high throughput. Linear Temporal Logic (LTL) verifies object consistency, automatically flagging semantic errors between frames, thus enabling automated detection of visual inconsistencies. These innovations collectively offer a unified solution for low-latency, high-precision generative editing across long-form video, while also providing correctness guarantees and practical deployment strategies that could be extended to other visual systems. Finally, the project will undergo rigorous system-level validation on both consumer-grade and datacenter-grade GPUs and will be released as open-source software with accompanying benchmark datasets and courses to broaden participation in generative AI. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.