Humanoid Robot Baseline Performance Benchmark

Proposal for a Comprehensive Method to Evaluate Minimum Expected Physical Capabilities for Humanoid Robots.

Background

In 2013-2014, NIST designed the tests for the DARPA Robotics Challenge (DRC) and assisted DARPA in implementing them during the competitions. The DRC was the last time humanoid robot performance was rigorously measured between robots. A new benchmark is needed to assess current humanoid robot capabilities across industry and academia, and to help steer future performance. 

Outline

The proposed Baseline Performance Benchmark is a low-footprint set of locomotion and manipulation tasks. These tasks are derived mostly from previously standardized NIST test methods with quantifiable and performance metrics. The tasks are intended to represent the minimum capabilities that one would expect commercially-available humanoid robots to be able to do (whether for industrial, home, healthcare, or other applications). The results will establish a comparable measure of capability between current industry-leading robots, as well as a common set of tasks manufacturers and researchers can use for further development of these robots.
 

Task Options:

This set of tasks establishes a set of common basic tasks as a baseline of performance for humanoids, while exercising:

• Domain-agnostic, basic humanoid robot mobility and manipulation/dexterity capabilities,
• Coordinated loco-manipulation capabilities combining mobility and manipulation tasks,
• Whole-body awareness and control through confined-space manipulation tasks, and
• Minimal reasoning, task and scene understanding, and decision making.

Credit: NIST

Credit: NIST

Benchmarking Process

The design of the baseline benchmark apparatus is being developed in collaboration with industry and the research community. Participation in this process is being sought to help achieve consensus on the tasks that should be included. NIST will fabricate a limited number of apparatuses to be freely distributed to participating U.S. humanoid robot manufacturers and established regional testing facilities. The designs and 3D models of the apparatus will also be published for use as a physical and/or virtual testbed for robot training and control development. 

Robot manufacturers who choose to have their robots tested will be able to elect to receive an apparatus (pending availability) and run their own tests, or to have their robots tested at NIST or at a participating facility. Test results will be collected under pre-approved data sharing agreements between NIST and the robot manufacturers in order to protect IP and any attribution if needed. Results will be aggregated to showcase the state-of-the-art of humanoid robot capabilities.
 

Considerations for Prospective Participating Robot Manufacturers and Collaborators

1. Do these tasks sufficiently exercise minimum humanoid capabilities?
2. Are there tasks that would better exercise whole-body-control and/or loco-manipulation?
3. What additional constraints would you need for testing your robots in such a suite of tasks?
4. Are you willing to help design the benchmark and/or have your robots tested?
5. Are you interested in being a participating test facility?

Contact Us

Please contact us if you would like to contribute answers to the above questions, to provide suggestions, have questions, or are interested in participating with the project. Points of contact are Dr. Benjamin Beiter (benjamin.beiter [at] nist.gov (benjamin[dot]beiter[at]nist[dot]gov)) and Dr. Kamel Saidi (kamel.saidi [at] nist.gov (kamel[dot]saidi[at]nist[dot]gov)).