Elements: DigitizedNYC: Large-Scale 4D Urban Digital Twin for Embodied AI

About This Grant

Urbanization is shaping the future of our world, with more than two-thirds of the global population projected to reside in cities by 2050. This significant challenge aligns with scientific innovations in embodied AI and smart cities, poised to revolutionize the way intelligent agents interact with the physical world and respond to the needs of varied and evolving urban communities. However, limitations in deployment range and the risks in real-world experiments hinder systematic developments and evaluations of emerging research in embodied AI. A promising pathway to address this gap is the creation of high-fidelity 4D digital twins of large-scale urban environments. This project supports research looking to develop DigitizedNYC, a realistic 4D digital twin of New York City that provides a shared platform for testing and comparing AI systems and smart city technologies. It helps researchers create repeatable experiments, making scientific studies more reliable and easier to build upon. This open and community-driven platform will promote reproducible science, support national infrastructure for embodied AI research, and advance fields such as autonomous mobility, urban planning, and accessibility. By integrating AR/VR tools, it will also provide immersive STEM education and outreach to students and professionals, contributing to a well-prepared scientific workforce. Many existing urban digital twins rely on aerial or indoor models that miss the complexity of real street-level environments, or on 2D videos that lack interactive capabilities. Creating accurate 4D digital twin models remains difficult due to visual challenges like dynamic lighting, weather, and moving objects, as well as technical issues such as sensor drift from long mapping trajectories. To fill this gap, this project performs research that provides a safe, realistic research cyberinfrastructure, accelerating the trustworthy development of embodied agents and smart city technologies. This project explores the following innovations: (1) Long-term multi-traversal mapping that will utilize consensus across multiple traversals to retain only consistent and permanent elements of the urban environment, while transient objects, such as pedestrians and vehicles, will be filtered out; (2) High-fidelity simulation will turn the large-scale 3D data into a 4D simulation that can be used for evaluation and even training of robot learning algorithms; (3) Science-driven cyberinfrastructure demonstration will show how DigitizedNYC serves as a city's important infrastructure that allows worldwide researchers and developers to easily test and fairly compare solutions for various embodied AI tasks, such as urban navigation. This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Division of Civil, Mechanical and Manufacturing Innovation within the Directorate for Engineering. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.