Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Performance Metrics for Evaluating Object and Human Detection and Tracking Systems

Published

Author(s)

Afzal A. Godil, Roger Bostelman, William P. Shackleford, Tsai H. Hong, Michael O. Shneier

Abstract

In this report, we provide an overview of various performance evaluation metrics for object detection and tracking for robot safety applications in smart manufacturing. We present four different types of performance evaluation metrics based on detection, tracking, perimeter intrusion, and motion tracking and pose estimation. The basis for comparing the strengths and weaknesses of different object detection and tracking algorithms is to evaluate their results on a set of tasks with known ground-truth data using the same performance metrics. The tasks, the ground-truth data, and performance evaluation metrics and test procedures can help vendors justify claims about the performance of their systems and assist users and manufacturers to compare systems for their particular automation tasks. They will also allow researchers to fully understand the strengths and limitations of different approaches. This is an essential step towards establishing the credibility of object detection and tracking for real time manufacturing and robotic applications. The performance metrics and evaluation methods are an essential first step towards providing scientific foundations for developing robot safety standards that enable the use of perception systems in manufacturing applications and particularly in providing confidence in systems to be used for safety-critical applications.
Citation
NIST Interagency/Internal Report (NISTIR) - 7972
Report Number
7972

Keywords

Manufacture, performance measurement, detection, tracking

Citation

Godil, A. , Bostelman, R. , Shackleford, W. , Hong, T. and Shneier, M. (2014), Performance Metrics for Evaluating Object and Human Detection and Tracking Systems, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.IR.7972, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=914820 (Accessed February 26, 2024)
Created July 30, 2014, Updated September 23, 2021