Perception for Everyday Human Robot Interaction

Abstract

The ability to build robotic agents that can perform everyday tasks heavily depends on understanding how humans perform them. In order to achieve close to human understanding of a task and generate a formal representation of it, it is important to jointly reason about the human actions and the objects that are being acted on. We present a robotic perception framework for perceiving actions performed by a human in a household environment that can be used to answer questions such as “which object did the human act on?” or “which actions did the human perform?”. To do this we extend the RoboSherlock framework with the capabilities of detecting humans and objects at the same time, while simultaneously reasoning about the possible actions that are being performed.