Benchmarking and Standardization of Intelligent Robotic Systems

Published

March 18, 2010

Author(s)

Rajmohan Madhavan, R Lakaemper, Tamas Kalmar-Nagy

Abstract

From mundane and repetitive tasks to assisting first responders in saving lives of victims in disaster scenarios, robots are expected to play an important role in our lives in the coming years. Despite recent advances in mobile robotic systems, lack of widely accepted performance metrics and standards hinder the progress in many application areas such as manufacturing, healthcare, and search and rescue. In this paper, we outline the importance of the development of standardized methods and objective performance evaluation/benchmarking of existing and emerging robotic technologies. We provide a survey of significant past efforts by researchers and developers around the globe and discuss how we can leverage such efforts in advancing the state-of-the-art. Using an example of designing a 'standard' evaluation toolkit for robotic mapping, we illustrate some of the problems faced in developing objective performance metrics whilst accommodating the requirements and restrictions imposed by the intended domain of operation and other practical considerations.

Proceedings Title

Proceedings of the 14th International Conference on Advanced Robotics (ICAR 2009), Munich, Germany, June 22 - 26, 2009.

Conference Dates

June 22-26, 2009

Conference Location

Munich

Pub Type

Conferences

Download Paper

Local Download

Keywords

benchmarking, standardization, intelligent systems, robotic systems, performance metrics, performance evaluation

Citation

Madhavan, R. , Lakaemper, R. and Kalmar-Nagy, T. (2010), Benchmarking and Standardization of Intelligent Robotic Systems, Proceedings of the 14th International Conference on Advanced Robotics (ICAR 2009), Munich, Germany, June 22 - 26, 2009., Munich, -1, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=904625 (Accessed October 15, 2025)