FRR: Learning Object-Centric Representations from Human Demonstrations for Robot Manipulation

About This Grant

Most robots today work in factories where people need to program every step of a task. This project explores how to make robots easier to use by helping them learn from watching people. This project uses artificial intelligence using reinforcement and imitation learning approaches. The goal is to teach robots how to see and handle objects by using video demonstrations of humans performing different actions. This is especially helpful with learning to grasp and move objects that have never been seen by the robot before. This way, robots can copy human tasks without needing detailed 3D models of every object they need to touch. If successful, the project could help bring robots into homes, hospitals, and other everyday places. It could also make robots more useful for people without the need for highly technical training. The project will support STEM education by building tools for teaching robotics and by including students in hands-on learning experiences. This project develops a new learning framework that allows robots to understand and imitate object-based manipulation tasks. The research team will design models that enable robots to recognize and segment objects from visual structured memory. Next the robots will learn where and how to grasp objects by observing human contact patterns. The robot will then generate motion plans to manipulate those objects for specific goals. Lastly it will plan sequences of actions to complete multi-step tasks. The robots will use video data of people performing tasks in the real world to train the models. This is helpful as it is replicable and does not rely on physical interaction with a person to train the robot. This approach aims to help robots generalize and adapt to new tasks, environments, and platforms. It combines perception and learning from demonstration, for objects that have not been modeled or seen before. The research contributes to the fields of robot learning, object perception, and manipulation planning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.