Collaborative Research: FRR: Enabling Robots with Adaptive Projector-Based AR in Dynamic Environments

About This Grant

This research explores projector-based augmented reality (AR) for communication from a robot. Because projected AR has traditionally required relatively flat surfaces to work, this novel approach allows people to view projected images and messages from robots in unstructured environments. In situations like search and rescue, concert halls, and cluttered homes, it can be hard to know what a robot's next actions might be. This approach supports users need to understand the robot's motions and intent when they are moving near people. This research enables robots to communicate by projecting images or messages onto non-flat surfaces. The project uses computer vision approaches to sense the environment and capture the geometry, photometry, and semantics at the same time. It is expected that this will improve communications when the robot is operating near people. This will also allow the robot to provide directions or point to objects without the need for special equipment. The project will do human studies to determine the effectiveness of the communications. This could make robots easier to communicate with in different situations in daily life. The project will directly advance the fields of Human-Robot Interaction, projected-AR, computer vision, and artificial intelligence (AI) by enabling robots to adaptively-project situated visualizations to multiple people at the same time. The proposed solution enables robots to project onto non-flat surfaces with flexible textures. Prior research has focused primarily on flat and mostly smooth surfaces. An adaptive projector-robot system will be created and evaluated to ensure people prefer and accept the system. The project uses unified computer vision techniques to understand an object’s geometry, photometry, and semantics at once. As the robot understands its environment, it can project onto objects that are not next to each other. The robot can dynamically adjust the image, so it compensates for textured surfaces. This is very important for the acceptance and use of robots in different environments. To make sure it works, the researchers will test different objects, scenarios, and locations (e.g., cluttered homes, lecture halls, and search and rescue scenes). The project will recruit participants to see whether they know what the robot is projecting and how to respond to the robot's communications. The project will compare different projection behaviors in real-world group settings. The goal is to measure the preferences and acceptance levels of different groups of people interacting with robots. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.