At New NIST Facility, Response Robots Must Measure Up

August 29, 2014

The growing number of visitors with mechanical escorts traveling to the newest building on the Gaithersburg, Md., campus of the National Institute of Standards and Technology (NIST) come expecting a disaster. And their hosts, members of NIST's Emergency Response Robots Project, never disappoint.

With careful attention to detail, lead engineer Adam Jacoff and his team have appointed much of the interior and exterior of NIST's year-old Robotics Test Facility with features inspired by catastrophe—earthquakes, hurricanes, explosions, nuclear-reactor meltdowns, and other cataclysmic events. Each suite—about three dozen in all—is designed for testing and measuring capabilities of robots built to assist search-and-rescue teams, bomb squads, and others who respond to natural and man-made disasters or other hazardous situations.

At this arena of disarray-by-design, robot manufacturers, first responders and military personnel, and researchers aiming for next-generation response-robot capabilities can determine how well and how reliably their particular machines are likely to carry out tasks at disaster scenes. For example, can a robot maneuver through rubble, scale a collapsed wall, or climb a ladder? Can it find victims trapped in wreckage, wield a hose, close a valve leaking toxic gas, operate power tools, or disable an improvised explosive device? Does the power source last as long as specified, and what is the range of the robot's communication system?

The 892-square-meter (9,600 square-foot) Robotics Test Facility is where many in the robotics and first-responder communities come for answers. At this proving ground, tests for measuring response-robot performance are prototyped, refined, and, ultimately, proffered for community-wide adoption as consensus standards.

"You need to measure how both the current practice of robotics and the state of the science and the emerging technology are addressing the particular challenges of disaster response," Jacoff explains. "This is best done in an environment that enables collaboration, facilitates adoption of best-in-class solutions, and helps the entire industry raise its sights and pursue the next set of advances that will lead to more capable systems."

The new facility is a big step up from the converted NIKE missile site where the performance-testing program got its start. Since 2005, 15 NIST tests have been adopted as standards by ASTM International and about 60 more are under various stages of development or review. The assortment of robots tested also has expanded beyond remotely operated ground vehicles to include aquatic vehicles and small unmanned aerial systems.

"Standard test methods are essentially just agreed-upon ways to test robotic capabilities," Jacoff says. "So we are not developing a specification for a 'standard robot' of any kind, like an equipment standard. Rather, we are developing a standard way to test remotely operated robotic systems so that performance can be measured and compared."

Realistic, reliable, and reproducible, these benchmark standards are credited with helping the young field of disaster robotics to make tangible progress since the technology was first deployed—at the collapse of the World Trade Center towers following the terrorist attacks on Sept. 11, 2001. Since then, robots have been used at more than 30 disaster sites around the world. The increase in deployments reflects improvements in capabilities, especially for reconnaissance tasks. But growing even faster is recognition of the many as-yet unrealized opportunities for robots to render essential, timely assistance in unpredictable hazardous environments. Getting there will require leaps in capabilities.

Response Robot Deployments, 2001 – 2013 

Year Event Robot Type* Year Event Robot Type*
2001 World Trade Center (USA) 4     2008 Hurricane Ike (USA)     1
2001 Jim Walter No. 5 Mine (USA) 1     2009 Cologne Building Collapse (Germany) 2    
2002 Barrick Gold Dee Mine (USA) 1     2009 L'Aquila Earthquake (Italy)   1  
2004 Brown's Fork Mine (USA) 1     2010 Haiti Earthquake   1 1
2004 Niigata Chuetsu Earthquake (Japan) 1     2010 Wangjialing Coal Mine (China) 1    
2004 Hurricane Charley (USA) 1     2010 Upper Big Branch Mine (USA) 1    
2004 Excel No. 3 Mine (USA) 1     2010 Deepwater Horizon (USA)     16
2005 DR No. 1 Mine (USA) 1     2010 Prospect Towers (USA) 2    
2005 McClane Canyon Mine (USA) 1     2010 Missing Balloonists (Italy)     1
2005 La Conchita Mudslides (USA) 1     2010 Pike River Mine (NZ) 2    
2005 Hurricane Katrina (USA) 1 3   2011 Christchurch Earthquake (NZ) 1 1  
2005 Hurricane Wilma (USA)   1 1 2011 Tohoku Earthquake (Japan) 3 1  
2006 Sago Mine (USA) 1     2011 Tohoku Tsunami (Japan)     9
2007 Midas Gold Mine (USA) 2     2011 Fukushima Nuclear Emergency (Japan) 7 2  
2007 Crandall Canyon Mine (USA) 1     2011 Naval Base Explosion (Cyprus)   2  
2007 I-35 Minneapolis Bridge Collapse (USA)     2 2011 Great Thailand Flood   2  
2007 Berkman Plaza II Collapse (USA) 2 1   2012 Finale Emilia Earthquake (Italy) 2 2  
*Robot Type and Number Deployed: (G)round, (A)erial, and (M)arine
SOURCE: Robin M. Murphy, Disaster Robotics 2014 (The MIT Press).

Today's robots, according to Gill Pratt, program manager at the Defense Advanced Research Projects Agency (DARPA), are like one-year-old children beginning to walk and interact with the world.

Not content to allow the field to linger in infancy, Pratt came up with the idea for the DARPA Robotics Challenge (DRC). The $2 million, winner-take-all competition is intended to "transform the field of robotics and catapult forward development of robots featuring task-level autonomy that can operate in the hazardous, degraded conditions common in disaster zones."

Early on, DARPA turned to NIST and asked it to lead development of the DRC tests. NIST's job was to develop a mock-up of a disaster scenario prompted by Japan's Fukushima Daiichi nuclear meltdown, caused by the 2011 Great East Japan earthquake and tsunami. With DARPA's input, Jacoff's team distilled lessons learned from the Fukushima Daiichi response into eight basic tasks and converted these tasks into benchmark performance tests.

Fukushima: Early Lessons to Learn and Signs of Progress to Note

Following a major earthquake, a 15-metre tsunami disabled the power supply and cooling of three Fukushima Daiichi reactors, causing a nuclear accident on 11 March 2011. All three cores largely melted in the first three days.

The disaster was overwhelming. The window of opportunity to avert cascading perils was within the first 24 hours after the accident. But radiation and contamination levels were so high at the debris-strewn site that people could not carry out effective countermeasures.

"So, that's where robots come into play," DARPA program manager Gill Pratt explained during a 2012 meeting. "Can we use robots to go where people can't go to help us damp down a disaster. So taking that as a problem to be solved at DARPA, we now [are taking] a look at what sort of technologies do we need."

Although not during the critical first 24 hours, robots ultimately did provide valuable assistance. One, dubbed Quince, was a repeat student at the NIST course. The brainchild of Satoshi Tadokoro, a professor of engineering at Tohoku University in Sendai, Japan, Quince was the only robot to make its way up to the second, third, fourth and fifth floors of the damaged reactor building.

Without that robot making it deep into the reactor building, "we could have had no way—besides suicide workers—of measuring radiation and checking valves and pipes on the second through fifth floor of Unit 2 of Fukushima," says Tadokoro. "The cool shutdown of the power plant would have been impossible or would have happened much later than without the robot's surveillance."

"I can say that my Quince robot would not exist if there were no NIST test methods," adds Tadokoro.

The internationally recognized robot designer goes further, crediting the NIST test course with helping to advance the entire field of search-and-rescue robots,

"Back at the time of 9/11, response robots were very simple—for example, a toy-sized tank with a camera," Tadokoro says. "Because of NIST's methods for testing and challenging them, however, response robots have year by year been improved so that they have autonomy, manipulation capability, more mobility, can find hidden victims very well, can climb up and down steep stairs, and so on."

The new Robotics Test Facility opened just seven months before the DRC trials, which were held Dec. 20-21, 2013, at Florida's Homestead-Miami Speedway. Almost immediately, the new building began to fill with an assortment of building materials. And soon thereafter, NIST staffers were sawing, hammering, fastening, and assembling them into what began to resemble a jungle gym for robots. A jumble of cinder-blocks seemingly strewn about the facility's rear exterior would become the route that competing robots traversed in DARPA's staged disaster. Inside, engineers and technicians readied other tests that would challenge robots to locate and shut off valves, wield a tool to break through a wall, and accomplish other DRC tasks.

By early December, the team was packing up the simulated disaster scenario for shipment to Florida. Like all of NIST's robot performance tests, the eight DRC tests were designed to be stacked on standard pallets and arranged for easy reassembly at what Jacoff called "one of the biggest robotics evaluations on Earth."

Of 16 U.S. and Japanese contestants, eight robot teams emerged from the December trials. These teams will go on to compete in the finals, to be held June 5-6, 2015, in Pomona, Calif. NIST and DARPA are now discussing plans for an even more rigorous round of performance challenges.

"The DRC Finals will provide more difficult, faster-paced tests designed to mimic an actual disaster scenario," Pratt explained in an article. And unlike the trials, the finalists will not be connected to tethers to deliver electricity to the robots, nor to safety lines to protect them from falling or otherwise damaging themselves.

NIST's work for DARPA epitomizes the collaborative ethos that Jacoff and team have cultivated since their earliest days of developing robot test methods.

"Our aim always has been to provide a reliable, useful testing resource for all parties across the robot development spectrum, from researchers to manufacturers all the way through to the end users," Jacoff says. "We measure success, capture failures, and deliver lessons learned. Through it all, we remain friends."

To date, more than 100 response robots have run the gauntlet of NIST test methods. Scores more have been tested when the NIST team takes its arena on the road for the annual RoboCupRescue competitions for robotics researchers, which Jacoff co-founded, or to other response robot evaluation exercises geared to emergency responders and soldiers.

Increasingly, operators of response robots come to the NIST course to test drive their mechanical assistance and to sharpen their skills. Today, response robots are controlled remotely by trained human operators, but they are gradually becoming more adept at performing basic navigational tasks unaided.

The goal is to advance to a more cooperative working relationship—called supervised autonomy—where robots perform many simple tasks on their own but continue to rely on humans for judgment and situational awareness. To help measure progress toward this goal, NIST has developed a tiered framework for evaluating how well response robots decide and perform on their own.

"Turning over more of the decision-making and workload to robots would allow non-experts to operate response robots, which could speed up response times significantly," explains Elena Messina, leader of the Manipulation and Mobility Group within NIST's Intelligent Systems Division. At Fukushima, for example, operators required several weeks of training before robots could be first deployed after the site was inundated by the tsunami.

"With greater autonomy, robots could be deployed in the immediate aftermath of a hazardous event, when lives still can be saved and when conditions are too dangerous for search and rescue teams to enter," Messina says.

Inspiration for NIST's tests comes from robot performance requirements gleaned from various user communities—bomb squads, search teams, military reconnaissance units, and others. Ideas for new requirements, however, also may spring from a breakthrough innovation that robotics researchers or manufacturers may introduce while running their newest creations through the paces on the test course.

Testing and training at NIST helps to ensure that users are realistic in their expectations—no robot has yet to excel in all tests—and, at the same time, provides a reality check for marketing claims. Typically, a robot will undergo 20 to 30 tests, chosen by a manufacturer, prospective customer, or researcher to establish benchmark capabilities for intended applications or missions. The combination of tests also helps in determining performance and cost trade-offs.

Test results can guide robot procurements as well as set tangible improvement targets that researchers and manufacturers can aim for in the next iteration of their technology.

Defense and civilian agencies alike consult the NIST test results to inform their purchase decisions. For example, the Fire Department of New York has been a frequent customer. Most recently, it used the NIST test course to scope out robot capabilities and refine mission requirements before investing in a robot platform to support hazmat operations.

"The Fire Department of New York City would like to thank you and your team for supporting the evaluation and procurement of our first robotic platform dedicated to search and hazmat operations. Your suite of DHS-NIST-ASTM International Standard Test Methods for Response Robots helped us to identify and refine our key mission requirements, assess available combinations of robotic capabilities and reliability, and perform acceptance testing to ensure it met our specifications. This included specifying and testing newly developed features which are critical for our intended missions, including the robot's pressurized chassis, thermal imaging camera, and advanced communications. We appreciate the extensive support provided by you and your team throughout this process."

– Nicholas Del Re, Chief in Charge, FDNY Haz-Mat

The Defense Department's Joint Improvised Explosive Device Defeat Organization used NIST performance-test results to guide a $32 million procurement of ultra-lightweight reconnaissance robots for tactical military operations. Similarly, the National Capital Region Bomb Squad Working Group, consisting of 15 federal, state, and local bomb squads, based recent robot purchases on NIST performance test results.

After using quantifiable data on robot capabilities to specify their robot purchases, several agencies also have required that robot vendors perform "acceptance testing" under the same NIST standard test methods before the machines can be put into service.

The NIST robot test course sprouted from a Department of Homeland Security (DHS) project to develop standardized methods to test and measure the abilities of urban search-and-rescue robots. In addition to DHS and the Department of Defense, the Department of Justice has also funded the NIST special program.

The value of these DHS-NIST-ASTM International Standard Test Methods is now multiplying as organizations around the world are replicating the test apparatuses, conducting their own evaluations, and collaborating with NIST to prototype and validate new tests. Similar test facilities in the U.S. include Southwest Research Institute in San Antonio, Texas. The newest to come on line is the New England Robotics Validation and Experimentation Center at the University of Massachusetts Lowell. It joins NIST-inspired courses in Japan and Germany. Next up is a bomb squad training facility in Perth, Australia.

In addition, scores of organizations have installed subsets of the NIST test methods to support development and testing of specific capabilities, leading to innovation and hardening of new robotic skills that will improve disaster-response capabilities and help to keep responders out of harm's way, according to Jacoff.

– written by Mark Bello

– Media Contact: Michael E. Newman, 301-975-3025


Created March 23, 2017, Updated August 02, 2018