Performance Metrics for Evaluating Object and Human Detection and Tracking Systems

Published

July 30, 2014

Author(s)

Afzal A. Godil, Roger Bostelman, William P. Shackleford, Tsai H. Hong, Michael O. Shneier

Abstract

In this report, we provide an overview of various performance evaluation metrics for object detection and tracking for robot safety applications in smart manufacturing. We present four different types of performance evaluation metrics based on detection, tracking, perimeter intrusion, and motion tracking and pose estimation. The basis for comparing the strengths and weaknesses of different object detection and tracking algorithms is to evaluate their results on a set of tasks with known ground-truth data using the same performance metrics. The tasks, the ground-truth data, and performance evaluation metrics and test procedures can help vendors justify claims about the performance of their systems and assist users and manufacturers to compare systems for their particular automation tasks. They will also allow researchers to fully understand the strengths and limitations of different approaches. This is an essential step towards establishing the credibility of object detection and tracking for real time manufacturing and robotic applications. The performance metrics and evaluation methods are an essential first step towards providing scientific foundations for developing robot safety standards that enable the use of perception systems in manufacturing applications and particularly in providing confidence in systems to be used for safety-critical applications.

Citation

NIST Interagency/Internal Report (NISTIR) - 7972

Report Number

7972

NIST Pub Series

NIST Interagency/Internal Report (NISTIR)

Pub Type

NIST Pubs

Download Paper

https://doi.org/10.6028/NIST.IR.7972

Local Download

Keywords

Manufacture, performance measurement, detection, tracking

Citation

Godil, A. , Bostelman, R. , Shackleford, W. , Hong, T. and Shneier, M. (2014), Performance Metrics for Evaluating Object and Human Detection and Tracking Systems, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.IR.7972, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=914820 (Accessed April 20, 2024)