Perception Performance of Robotic Systems

Summary

Sensors and algorithms are essential elements of any robotic system that is meant to be safe, adaptive, and productive in manufacturing environments. In the past few years, there has been significant growth in sensor types and an increased diversity of corresponding algorithms that are used for perception (i.e., extracting information and knowledge from the raw sensor data). This makes it difficult for researchers, users, systems integrators, and even robot manufacturers to identify the right solutions to pair with a robot for a given implementation. This project will help to develop new industry standards by identify existing and emerging sensing and perception systems and developing a roadmap of needed standards to address performance gaps.

Past Events: NIST at the 2023 Automate Show

Workshop on Standards for 3D Perception Systems for Robotic Assembly Applications

NIST and ASTM E57 at the 2022 Automate Show

Description

To develop measurement science for sensing and perception system performance characterization to reduce the risk related to the adoption of new technologies and to advance the agility, safety, and productivity of collaborative industrial and mobile robots.

What is the Technical Idea?

Sensors and algorithms are essential elements of any robotic system that is meant to be safe, adaptive, and productive in manufacturing environments. In the past few years, there has been significant growth in sensor types and an increased diversity of corresponding algorithms that are used for perception (i.e., extracting information and knowledge from the raw sensor data). This makes it difficult for researchers, users, systems integrators, and even robot manufacturers to identify the right solutions to pair with a robot for a given implementation. The technical idea is to:

1. identify existing and emerging sensing and perception technologies and related software in order to understand the product and research solutions landscape; and
2. identify available standards and understand manufacturing users’, system integrators’, and solution providers’ (current and potential) use cases and performance requirements (including environmental conditions under which the sensing and perception systems must operate). This is crucial to mapping the performance gaps and to inform the development of metrics and test methods. These will contribute towards new industry standards that can address the performance gaps.

What is the Research Plan?

The conditions under which a sensor operates affect its performance. Examples of such environmental factors include lighting (intensity, color, direction, etc.), clutter, and scene type (indoor, outdoor, etc.). Characterization of environmental conditions and their impact on sensing and perception system performance is crucial to understanding how these systems work.

This project will help develop means of

• identifying which environmental conditions should be considered;
• controlling environmental conditions for conducting experiments;
• classifying or quantifying each of the relevant conditions; and
• measuring the impact of the environmental conditions on the performance of different sensor types and perception algorithms.
• Manufacturing parts pose particular challenges for most sensor systems due to their material specularity and dearth of surface features that are useful for many perception algorithms. Properties that will be investigated include surface finish, material, texture, shape, and reflectivity. The project will explore the relationship between a part’s properties and their impact on a perception system’s performance.
• The ability to successfully identify an object or its pose depends on the geometry of the part and the pose itself. This project will develop mathematical formulations and other methods to predict how well algorithms can identify an object and calculate its pose from limited data. The classic example of this is partially-occluded objects in bins.
• A robot performing an assembly operation may require lower uncertainties about the position of the target location than a sensor can provide. This project will develop methods for modeling the uncertainty of sensing and perception systems and for propagating the uncertainty to the robot’s intended manufacturing tasks. This will build on work begun at NIST on modeling the impact of sensor uncertainties on task-specific results.
• This project will also develop test artifacts, procedures, and analytical tools to assess performance characteristics of a range of sensor systems. Sensor datasets can be provided to enable assessment of perception algorithms. For example
  • datasets will be collected from a variety of sensors under controlled conditions, with ground truth; and
  • the feasibility of using synthetic data from computer-aided design (CAD) models for testing perception systems and algorithms will be investigated.
• Understanding the required quality of data and images produced from CAD models will help guide progress towards NIST-validated datasets that are disseminated to facilitate algorithm evaluation. Datasets may also be useful for perception algorithms based on machine learning, especially if large numbers can be generated synthetically. Hence, this project may support the Embodied AI and Data Generation effort.
• Sensor performance degradation is a significant challenge in manufacturing environments, often leading to costly operational failures. This project will develop models and methods that detect health deterioration, such as loss of calibration, to enable perception systems to perform self-monitoring.