Performance Metrics and Test Arenas for Autonomous Mobile Robots

Objective

The RoboCupRescue Robot League is an international league of teams with one objective: Develop and demonstrate advanced robotic capabilities for emergency responders using annual competitions to evaluate, and teaching camps to disseminate, best-in-class robotic solutions. 

Approach  

The league hosts annual competitions to 1) increase awareness of the challenges involved in deploying robots for emergency response applications such as urban search and rescue and bomb disposal, 2) provide objective performance evaluations of mobile robots operating in complex yet repeatable environments, and 3) promote collaboration between researchers. Robot teams demonstrate their capabilities in mobility, sensory perception, localization and mapping, mobile manipulation, practical operator interfaces, and assistive autonomous behaviors to improve remote operator performance and/or robot survivability while searching for simulated victims in a maze of terrains and challenges. Winning teams must reliably perform 7-10 missions of 20-30 minutes each from various start points to find the most victims. As robots continue to demonstrate successes against the obstacles posed in the arenas, the level of difficulty will continually be increased so the arenas provide a stepping-stone from the laboratory to the real world. Meanwhile, the annual competitions provide direct comparison of robotic approaches, objective performance evaluations, and a public proving ground for capable robotic systems that will ultimately be used to save lives. 

Competition Vision  

When disaster happens, minimize risk to search and rescue personnel while increasing victim survival rates by fielding teams of collaborative mobile robots which enable human rescuers to quickly locate and extract victims. Specific robotic capabilities encouraged in the competition include the following:  

• Negotiate compromised and collapsed structures
• Locate victims and ascertain their conditions
• Produce practical sensor maps of the environment
• Establish communications with victims
• Deliver fluids, nourishment, medicines
• Emplace sensors to identify/monitor hazards
• Mark or identify best paths to victims
• Provide structural shoring for responders

These tasks are encouraged through challenges posed in the arena, specific mission tasks, and/or the performance metric. Demonstrations of other enabling robotic capabilities are always welcome.

Search Scenario

A building has partially collapsed due to earthquake. The Incident Commander in charge of rescue operations at the disaster site, fearing secondary collapses from aftershocks, has asked for teams of robots to immediately search the interior of the building for victims. The mission for the robots and their operators is to find victims, determine their situation, state, and location, and then report back their findings in a map of the building with associated victim data. The section near the building entrance appears relatively intact while the interior of the structure exhibits increasing degrees of collapse. Robots must negotiate and map the lightly damaged areas prior to encountering more challenging obstacles and rubble. The robots are considered expendable in case of difficulty.

Field Description  

The RoboCupRescue arenas constructed to host these competitions consist of emerging standard test methods for emergency response robots developed by the U.S. National Institute of Standards and Technology through the ASTM International Committee on Homeland Security Applications; Operational Equipment; Robots (E54.08.01). They are repeatable test method apparatuses that anybody can build and practice. The competition field is divided into color-coded arenas that form a continuum of challenges with increasing levels of difficulty for robots and operators and highlight certain robotic capabilities: 

• Simulated Victims: Simulated victims with several signs of life such as form, motion, head, sound and CO2 are distributed throughout the arenas requiring directional viewing through access holes at different elevations.
• Yellow Arena: For robots capable of fully autonomous navigation and victim identification, this arena consists of random mazes of hallways and rooms with continuous 15° pitch and roll ramp flooring.
• Orange Arena: For robots capable of autonomous or remote teleoperative navigation and victim identification, this arena consists of moderate terrains with crossing 15° pitch and roll ramps and structured obstacles such as stairs, inclined planes, and others.
• Red Arena: For robots capable of autonomous or remote teleoperative navigation and victim identification, this arena consists of complex stepfield terrains requiring advanced robot mobility.
• Blue Arena: For robots capable of mobile manipulation on complex terrains to place simple block or bottle payloads carried in from the start or picked up within the arenas.
• Black/Yellow Arena (RADIO DROP-OUT ZONE): For robots capable of autonomous navigation with reasonable mobility to operate on complex terrains.
• Black Arena (Vehicle Collapse Scenario): For robots capable of searching a simulated vehicle collapse scenario accessible on each side from the RED ARENA and the ORANGE ARENA.
• Aerial Arena (< 2 KG, VTOL sUAS): For small unmanned aerial systems under 2 kg with vertical take-off and landing capabilities that can perform station-keeping, obstacle avoidance, and line following tasks with varying degrees of autonomy.

Qualification Process

The RoboCupRescue Robot League encourages participation by interested organizations from around the world, but limits participation to one team per organization. Regional open competitions are hosted around the world to provide essential practice deployment experience for teams and to support qualification of teams for the World Championship competition each year. The World Championship includes a maximum of 5 teams from each regional open competition, which are typically the three awardees and up to two more teams based on the previous year's awards, Best-In-Class demonstrations in the current year, and other considerations noted below.  

RoboCupRescue Robot League regional open competitions are hosted in Japan, Germany, Iran, Thailand, Mexico, and we are actively trying (but need help from locals) to establish regional open competitions in other areas as well. All teams should use regional open competitions to:  

• Practice your deployment strategies,
• Familiarize yourselves with the arenas and rules, and
• Demonstrate your capabilities to the Technical Committee.

Unfortunately, due to the close scheduling of the regional open competitions and the World Championship each year, the qualification process can't be completely sequential because teams sometimes have trouble making it to the World Championship on short notice. So the Technical Committee qualifies an initial set of teams each year for the World Championship based on performance in the previous year's competition and their updated team description papers. Teams that win a 1st, 2nd, or 3rd place award at any regional open competition and submit a team description paper are qualified for the World Championship competition in that year and the following year.

The Technical Committee usually keeps a few slots open in the World Championship competition to include additional capable teams that emerge from the regional open competitions each year. When there is no regional open competition in the region of the World Championship, the technical committee typically qualifies proportionally more teams from the local region, while encouraging them to try to participate in other regional open competitions as well. At large qualifications may also be granted for teams in regions without a regional open competition. Although it is clear that the best route to participate in a World Championship competition is to perform well in a regional open competition and write a good team description paper. You will find the lessons learned from both are well worth the effort.

Each year the Technical Committee looks at the following criteria to qualify teams for the World Championship:  

• Your TDP which describes improvements to your robot based on lessons learned from the previous year's competitions that produced a semi-final round appearance in the World Championship, a Best-In-Class award in the World Championship, or a Place award in a regional open.
• Your TDP describes a particularly interesting or innovative approach that the Technical Committee considers likely to perform well at the World Championship competition even without previous experience at a regional open competition, especially when there is not yet a regional open competition nearby to you. This is especially possible if you can demonstrate your capabilities convincingly within representative arena apparatuses through video of the robot performing any or all of the requisite capabilities:
  • advanced mobility (traversing random stepfields or confined space cubes)
  • navigation (wall following, centering between obstacles or constrictions)
  • localization and mapping (2D/3D maps, SLAM on non-flat-flooring i.e. pitch/roll ramps, low-profile stepfields)
  • directed perception (visual acuity for near/far/dark/light, sensor probing into voids, sometimes with reaching)
  • victim identification (fusion of the various sensory signals to improve confidence and reduce errors)
  • autonomy (assistive features, bounded intervals, or fully autonomous performance of any or all of the above)
  • effective operator interfaces

Finally, some qualifications for the World Championship may be granted by the Technical Committee to include particular countries in the league, encourage technologies that the league should be investigating, or to support other league outreach efforts.

Trustees

• Adam Jacoff, National Institute of Standards and Technology, USA (Trustee Emeritus)
• Satoshi Tadokoro, Tohoku University, Japan (Trustee Emeritus)

Executive Committee

• Johannes Pellenz, University of Koblenz-Landau, Germany (Exec 2013-2016)
• Jafar Chegini,Iran (Exec 2015-2018)
• Jackrit Suthakorn, Mahidol University, Thailand (Exec 2015-2017)
• Raymond Sheh, Australia (Exec 2015-2017)
• Andreas Birk, International University Bremen, Germany (Exec Emeritus)
• Tetsuya Kimura, Nagaoka Univ. of Technology, Japan (Exec Emeritus)

Technical Committee

• Gerald Steinbauer, Austria (TC 2012-2015)
• Tetsuya Kimura, Nagaoka Univ. of Technology, Japan (TC 2015-2017)
• Masayuki Okugawa, Japan (TC 2015-2018)
• Sören Schwertfeger, Germany (Emeritus)

Organizing Committee

• Stefan Kohlbrecher, Local Co-Chair
• Johannes Pellenz, Local Co-Chair
• Ann Marie Virts, National Institute of Standards and Technology, USA (OC Chair)
• Adam Jacoff, National Institute of Standards and Technology, USA
• Sören Schwertfeger
• Nico Kroll
• Ehsan Mihankhah
• Kenneth Kimble, National Institute of Standards and Technology, USA