Benchmarking Functionalities of Domestic Service Robots Through Scientific Competitions

Abstract

Benchmarking via carefully designed competitions makes it possible to provide a common framework for the rigorous comparison of intelligent and autonomous systems; competitions may play the role of scientific experiments while being appealing both to researchers and to the general public thus promoting critical analysis of systems outside the labs. This paper describes our approach to benchmarking domestic service robots through organizing recurrent competitions under the European Robotics League. It details the tools and benchmarks designed to evaluate the performance of robots at task and functionality levels. In particular, the functionality benchmarks for object perception and navigation are described and an overview of the new benchmarks to appear in the league is presented.