Collaborative Research: NeTS: Small: Revolutionizing Immersive Video Streaming with Neural Content Representation

About This Grant

This collaborative project explores immersive video streaming technology that lets people see and interact with lifelike 3D environments. Instead of watching a regular video on a flat screen, users can move around a scene, change their viewpoint, and experience the environment from different angles in real time. This is made possible through advanced techniques that simulate how light travels through a scene using artificial intelligence. By leveraging computer vision, computer graphics, and machine learning, the project aims to make these 3D experiences smooth, detailed, and fast, even over current Internet connections. This project builds a high-performance immersive video streaming system using neural radiance fields, an advanced 3D scene representation powered by machine learning. It is structured around three key thrusts: improving resiliency, optimizing bandwidth, and reducing latency. Resiliency is addressed through lightweight models that recover lost content, adaptive encoding based on content robustness, and artifact reduction via reprojection. Bandwidth is optimized by filtering less important content, prioritizing semantically significant regions, and applying hybrid upsampling. Latency is reduced through intelligent packet ordering, caching intermediate results, and collaborative edge-client rendering. These innovations are integrated into a unified framework called NeuVol, which will be evaluated on diverse hardware and real-world network conditions. The project’s broader impacts include both technological innovation and societal advancement. It will enhance immersive applications in fields such as remote training, collaborative design, and medical diagnostics by enabling efficient delivery of high-quality 3D content. Educationally, it will integrate research outcomes into courses on multimedia systems and immersive computing, offering students hands-on experience with emerging technologies. The project will actively engage undergraduate researchers through active mentorship. Public outreach will feature demonstrations at schools, community centers, and science fairs to promote awareness of immersive technologies and inspire interest in engineering and computer science among the broader public. The project brings together researchers from George Mason University and the University of Illinois Urbana-Champaign to design, build, and test NeuVol and make it available to a wide audience through open research. All research artifacts developed through this project will be made publicly available via a dedicated project website at https://neuvol.github.io. The website will be actively maintained throughout the project duration and archived for continued access beyond the project’s conclusion. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.