Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Test Methods and Metrics for Effective HRI in Collaborative Human-Robot Teams

Human Awareness Evaluations

Megan Zimmerman from NIST performs metrology tests of a human-aware vision system for robot situational awareness.

Workshop Report

The need for effective human-robot interaction (HRI) continues to present challenges for the field of robotics.  As new technologies are integrated into human-robot teams in a myriad of application domains, exposures to and expectations from robots are growing rapidly.  A key factor that limits the success of human-robot teams is the lack of principled metrics for assessing the effectiveness of HRI.

The necessity for validated test methods and metrics for human-robot teaming is driven by the desire for repeatable and consistent evaluations of HRI methodologies.  Such evaluations are critical for advancing underlying models of HRI, as well as establishing traceable mechanisms for vendors and consumers of HRI technologies to assess and assure functionality.

This full-day workshop will address the issues surrounding the development of test methods and metrics for evaluating the performance of human-robot teams across the multitude of human-centered application domains, including industrial, social, medical, field and service robotics.  This workshop will focus on establishing the diversity of approaches for addressing HRI metrology in different robotic domains, and identifying the underlying issues of traceability, objective repeatability, and transparency in HRI metrology.

The workshop organizers are seeking full paper submissions for presentation.  Papers should be limited to 6 pages in the IEEE Proceedings format.  Accepted papers will also be invited to submit their papers to a special journal issue to be published at a later date.  Extended abstracts and posters will also be accepted for consideration for presentation during a poster session.

Solicited paper topics include:

  • Test methods and metrics for evaluating human-robot teams
  • Best practices and real-world case studies in human-robot teaming
  • HRI data set generation, formatting, and dissemination for human-robot teams
  • Verification and validation of HRI studies involving human-robot teams
  • Benchmarking performance of human-robot teams
  • Evaluation of novel human-robot team designs and methods

Important Dates:

  • Paper submission deadline: Friday, 8 February, 2019
  • Notice of acceptance for presentation: Friday, 15 February, 2019
  • Submission of posters and extended abstracts for presentation: Friday, 8 February, 2019
  • Notice of acceptance of posters for presentation: Friday, 15 February, 2019
  • Workshop date: Monday, 11 March, 2019


  • jeremy.marvel [at] (Dr. Jeremy A. Marvel), National Institute of Standards and Technology (NIST), USA
  • shelly.bagchi [at] (Shelly Bagchi), National Institute of Standards and Technology (NIST), USA
  • megan.zimmerman [at] (Megan Zimmerman), National Institute of Standards and Technology (NIST), USA
  • murat.aksu [at] (Murat Aksu), National Institute of Standards and Technology (NIST), USA
  • brian.antonishek [at] (Brian Antonishek), National Institute of Standards and Technology (NIST), USA
  • yue6 [at] (Dr. Yue Wang), Clemson University
  • ross [at] (Dr. Ross Mead), Semio
  • terry.fong [at] (Dr. Terry Fong), National Aeronautics and Space Administration (NASA), USA
  • hbenamor [at] (Dr. Heni Ben Amor), Arizona State University



Arizona State University Logo
Arizona State University


Clemson University
Clemson University








0900 – 0915

Welcome and Introduction

Megan Zimmerman

0915 – 1000


Dr. Laurel Riek

1000 – 1030

Contributor Presentations

Malte Jung

Kathleen Belhassein & Guilhem Buisan

1030 – 1050



1050 – 1135

Invited Talk

Tathagata Chakraborti

1135 – 1230

Contributor Presentations

Raquel Oliveira

Dan Szafir

José Lopes

1230 – 1300

Recap and Discussion


1300 – 1400

Lunch Break


1400 – 1445

Panelist Presentations

Yun Kyung Kim

Chung Hyuk Park

Megan Zimmerman

Katrin Lohan

1445 - 1530

Metrics Panel Discussion

Moderator:  Murat Aksu

1530 – 1550



1550 – 1605


Shelly Bagchi

1605 – 1650

Invited Talk

Dr. Seungbin Moon

1650 – 1730

Open Discussion, Recap, and Next Steps






Remote participants:

For those who wish to participate, but are unable to attend in person, a Google Meet has been scheduled to live-stream the workshop.


‎Dial In: (‪US)‪+1 559-854-1451

PIN: ‪630 283 416#


Click here to add the stream to your calendar.

Created December 12, 2018, Updated October 5, 2021