Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

2011 Solutions in Perception Challenge Performance Metrics and Results



Jeremy Marvel, Tsai H. Hong, Elena R. Messina


The 2011 Solutions in Perception Challenge (SPC) presented an international collection of teams with the opportunity to develop algorithms that could positively identify and accurately locate in space an arbitrary collection of artifacts. Researchers at the National Institute of Standards and Technology (NIST) generated a series of artifacts synonymous with industrial parts, a modular fixturing system capable of accurately and precisely positioning the artifacts within a 6 degree of freedom (6DOF) work volume, and a relative pose scoring metric to quantify an algorithm's performance. Competing teams were presented with training and validation training sets consisting of red-green-blue (RGB) color images and 3D point cloud data of the artifacts. The top performers achieved over 70% accuracy in translation and pose estimation. In this paper we discuss the design of NIST's contributions, and present the teams' results from the 2011 SPC.
Proceedings Title
2012 Performance Metrics for Intelligent Systems Workshop
Conference Dates
March 20-22, 2012
Conference Location
College Park, MD, US


Ground Truth, 6DOF Metrology, Laser Tracker, Fixtures


Marvel, J. , Hong, T. and Messina, E. (2012), 2011 Solutions in Perception Challenge Performance Metrics and Results, 2012 Performance Metrics for Intelligent Systems Workshop, College Park, MD, US (Accessed July 24, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created March 21, 2012, Updated October 12, 2021