Performance of Human-Robot Interaction

Summary

With an increased interest in the reshoring of manufacturing to the United States, more businesses are turning to technology to accommodate the increased demand for American-made parts and products. The progression to full automation, however, is both long and extremely expensive. To bridge the gap between current manufacturing capabilities and future production goals, manufacturing processes must leverage both human and machine labor, often without sufficient lead time to provide adequate training for the use and maintenance of equipment. As such, a renewed interest in human-focused robotics and automation design, control, and programming has emerged. The Performance of Human-Robot Interaction project will expand on its prior work to produce novel measurement science to assess and assure the usability, performance, and trustworthiness of emerging robotic systems. Focus areas will include 1) performance and design considerations of novel interface designs for task-driven, user interfaces and experiences; 2) ethics and trust in human-facing robotic solutions; 3) the utility of digital twins for the purpose of human-oriented situation awareness and effective human-robot interaction; and 4) the design and evaluation of intrinsically safe, flexible materials for interaction and process evaluations. The outputs of this project – datasets, benchmarking tools, test methods, protocols, metrics, and standards – will enable integrators and end-users to maximize the effectiveness and efficiency of collaborative human-robot teams in production processes, and will benefit all scales of manufacturing that employ robot-assisted, skilled labor.

Description

Credit: Earl Zubkoff

Objective
To deliver a suite of datasets, benchmarking tools, test methods, protocols, metrics, standards, and information models to enable effective, human-robot collaboration in manufacturing, and advance interactive robot technologies to facilitate the safe and efficient teaming of people and robots that maximally leverages the strengths and capabilities of each toward meeting production goals.

Technical Idea
To achieve the specified objective, the Performance of Human-Robot Interaction project will focus on developing a collective metrology suite consisting of datasets, benchmarking tools, test methods, metrics, systems models, software libraries, and algorithms to evaluate the capabilities and performance of human-robot interfaces, mechanisms of trust and ethics in human-facing robots, and system and situation awareness models leveraged in human-robot interaction. This collective metrology suite will enable technology developers, integrators, and end users of collaborative robot technologies to:

• Integrate, evaluate, and optimize metrology technologies (i.e., sensors and algorithms) built into and influencing the design of interfaces intended to optimize the user experience. The project will develop test methods and metrics, labeled datasets, guidelines and best practices for HRI studies and dataset generation and publication, and virtual models that can be used to assess and assure effective information sharing and situation awareness.
• Assess and optimize the actions of robot systems and presentation of robot feedback to encourage trust and establishing common ground with human operators in collaborative tasks. The project will develop guidelines and documented best practices for the design of physical appearances, behaviors, and feedback mechanisms that encourage ease-of-use and integration and minimize perceived robot intent ambiguity for users working with and around robots.
• Drive innovation in the design and testing of systems that enable ethical interactions and implementations, establish and maintain trust in the capabilities, safety, and security of operator interests, and reinforce good faith relationships between stakeholders and machines. The project will develop guidelines and best practices that guide ethical robotic system design and implementation.
• Develop new behaviors and capabilities of interactive robot systems to support new operator training, team adaptability to process change and uncertainty, and responsiveness and utility in high-impact situations. The project will develop new protocols and test methods for the evaluation of human preferences and experience, as well as mechanisms for evaluating the effectiveness of information sharing protocols for efficient communication.
• Share and receive instructions, status updates, and diagnostic data using methods intuitive to both sides of the collaborative human-robot team using this project’s resulting models, metrics, and software libraries. The project will develop new test methods and software tools for assessing communication efficacy through advanced interfaces such as digital twins and wearable technologies, which can be used to optimize the presentation of task-relevant information to human operators.
• Assess and assure the efficiency and effectiveness of employing collaborative robots in flexible factory environments such that the costs and benefits of integrating robots into collaborative teams are optimally balanced in favor of the end user. The project will continue to develop new metrics, test methods, and standards for the evaluation of the impacts of employing collaborative robots in manufacturing environments (including the ease of integration, the cost of programming, impacts on team process performance, and the impacts on risk assessments and safety protocols).
• Employ robots capable of automatically and safely adapting to user preferences, experience, attention, and actions relevant to the collaborative task. The project will develop new test methods and metrics for the evaluation of human-aware collaborative robots, including metrology for the evaluation of operator intent and motions as they pertain to safe and effective human-robot collaboration.
• Leverage human-robot teams to safely collaborate to jointly handle flexible materials. The project will continue research in soft robotics and develop new metrology tools and artifacts for the joint handling of non-rigid objects and artifacts to be used to assess and assure the collaborative performance of human-robot teams.

Research Plan
The research plan focuses on four principal capabilities of robot systems that collectively contribute to human-robot integration and teaming: 1) creating models of trust and ethical human-robot interaction; 2) developing test methods, metrics, and best practices for the design of interfaces; 3) designing metrology tools for measuring the impacts and requirements of digital twins; and 4) developing novel metrology tools for soft robotics and flexible materials. Collectively, these focus areas comprise a collective suite of test methods, metrics, and protocols to assess and assure close-proximity human-robot interaction performance in emerging manufacturing applications. For each phase of development, the test methods, metrics, and protocols will be evaluated using the NIST human-robot interaction laboratory.

• Trust and Ethical Human-Robot Interactions in Manufacturing: As a skilled human workforce is expected to leverage robotic technologies to achieve ever-increasing demands for throughput and quality, considerable trust must be placed in the robotic systems. The safety, security, and privacy of related interactions are paramount to maintaining this trust, and the scientific research that feeds into commercial systems must maintain high ethical standards. The project will deliver protocols and best practices for the establishment of standardized guidelines for conducting human-robot interaction research, protecting sensitive information and demographics, and maintaining confidence and trust in daily interactions. Misapplications of artificial intelligence undermine the greater public trust in intelligent systems, so these procedures and protocols are critical for establishing and maintaining the trust of the operators who must routinely interact with these robotic systems.
• Designs of Interfaces: The project will deliver protocols and test methods for assessing the design and use of robot appearance and behavior in collaborative human-robot teaming. These test methods will quantify the impacts of the robot’s interfaces and means of interacting with human operators, and assess the effectiveness and efficiency of interfaces on the humans’ ability to program, control, and diagnose robotic applications.
• Impacts and Requirements of Digital Twins: Digital twins has grown steadily over the past four decades and new capabilities as a tool for designing and commissioning entire workcells in a factory environment have been developed. The project will continue research into the utility of digital twins as a human-robot interaction tool for offline design, commissioning, and programming, as well as a mechanism by which situation awareness can be established at multiple levels of operations (i.e., from factory-wide to individual processes). New protocols and tools will be developed in line with other research efforts toward user interfaces to better understand and predict the impacts on human operators, as well as the influences of information bandwidth, quality, fidelity, and means of presentation on the establishment and maintenance of situation awareness and effective communications. 

Recent Major Accomplishments

• NIST-developed reporting criteria have been adopted by the Association for Computing Machinery (ACM) for their annual HRI conference.
•  NIST-led standards efforts on HRI through IEEE have increased growth of stakeholder interest in HRI benchmarks, research replicability, and reporting criteria.
• NIST’s work to raise awareness of metrological challenges in HRI through publications (including special issues of archival research journals; book chapters on measurement science, verification, and validation of HRI; and numerous publications on HRI test methods and metrics) have resulted in significant community support for replicable and repeatable research, HRI applications- and technology-focused events, and improved reporting criteria for human-subject HRI studies.