Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Using Competitions to Advance the Development of Standard Test Methods for Response Robots



Adam S. Jacoff, Raymond Sheh, Ann Virts, Tetsuya Kimura, Johannes Pellenz, Soren Schwertfeger, Jackrit Suthakorn


Competitions are an effective aid to the development and dissemination of standard test methods, especially in rapidly developing, highly varied fields such as Urban Search and Rescue robotics. By exposing the development process to highly developmental systems that push the boundaries of current capabilities, it is possible to gain an insight into how the test methods will respond to the robots of the future. The competition setting also allows for the rapid iterative refinement of the test methods and apparatuses in response to new developments. For the research community, introducing the concepts behind the test methods at the research and development stage can also help to guide their work towards the operationally relevant requirements embodied by the test methods and apparatuses. This also aids in the dissemination of the test methods themselves as teams recreate them in their own laboratories and re-use them in work outside the competition. In this paper, we discuss how international competitions, and in particular the RoboCupRescue Robot League competition, have played a crucial role in the development of standard test metrics for response robots as part of the ASTM International Committee of Homeland Security Applications; Operational Equipment; Robots (E54.08.01). We will also discuss how the competition has helped to drive a vibrant developer community towards solutions that are relevant to first responders.
Proceedings Title
2012 Performance Metrics for Intelligent Systems Workshop
Conference Dates
March 20-22, 2012
Conference Location
College Park, MD, US


performance metrics, response robots, competitions, mobile robots, autonomous robots, manipulation, field robotics


Jacoff, A. , Sheh, R. , Virts, A. , Kimura, T. , Pellenz, J. , Schwertfeger, S. and Suthakorn, J. (2012), Using Competitions to Advance the Development of Standard Test Methods for Response Robots, 2012 Performance Metrics for Intelligent Systems Workshop, College Park, MD, US, [online], (Accessed May 25, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created May 16, 2012, Updated October 12, 2021