Embodied AI and Data Generation for Robotics

Summary

Credit: NIST

Upcoming Workshop - March 11-12, 2025

Workshop on Standards for Robot Absolute Accuracy and Performance Assessment | NIST

Artificial intelligence (AI) in the form of advanced machine learning models has been widely adopted in the technology world today. These machine learning models and techniques show great potential for improving the performance of robotic systems but pose unique challenges to both the machine learning and robotics aspects of the system. This is especially true in the manufacturing domain, where the cost, safety, and productive output of the robotic systems are of high importance. Currently, a large gap exists between embodied AI seen in academic research and what is feasible to implement in the real-world by manufacturers and integrators. The goal of this project is to help bridge this gap by implementing and characterizing various embodied AI approaches that are relevant and practical to manufacturing robotic system developers. As part of this process, the project will develop test methods assessing the performance of AI-enabled robotic systems and communicate these results to industry via guidelines, standards, reports, publications, and datasets. The manufacturing robotics community stands to benefit from this work by gaining new methods for assessing the feasibility of AI-based solutions to industry challenges, and the academic community stands to benefit through access to industry collaborators to facilitate the practical adoption of their research. NIST is uniquely qualified to address this because of our experience with robotic system measurement and test method development, as well as our wide range of contacts across industry and academia for facilitating technology transfer and collaborations.

Description

Objective
To facilitate the adoption of AI-based robotic approaches in practical manufacturing scenarios by creating test methods that target AI-enabled robotic systems, evaluating the performance of AI-enabled robotic systems, and creating manufacturing-relevant and AI-centric datasets.

Technical Idea
The technical idea for this project is to identify and characterize the challenges associated with adopting AI technologies in manufacturing robotic systems. Generally, manufacturers and integrators use automation (via robots) to improve productivity with respect to some key metrics (yield, cycle time, overall equipment effectiveness, etc.). This makes adopting emerging technologies difficult due to the uncertainties in estimating the impact of any new technological improvement on production. This is especially true for AI because implementing and evaluating the risks of an AI system has limited overlap with existing automation practices. To address this problem, manufacturers and integrators need a streamlined way of assessing the productive impact of AI systems, which NIST can provide through AI-specific productivity metrics and test methods.

To develop these AI metrics, we can leverage the fact that most AI applications follow a common pipeline involving data collection, data pre-processing, training, and deployment. While the particulars of each step may be different depending on the algorithm and application, the costs incurred at each step can be concretely defined and characterized. Furthermore, the productive benefits of the AI system can also be measured with the help of general machine learning metrics such as model accuracy, precision/recall, and mean average precision (mAP).

The technical challenge of developing these metrics lies in understanding the relationship between AI algorithms, robotic systems, and tasks, as well as their combined effects on cost and performance. Because there are countless ways to combine these three elements, it is crucial to consider a representative set of AI-enabled robotic systems to make sure that any metrics developed will adequately cover the range of possible data collection modalities, preprocessing/quality assurance methods, machine learning algorithms, training/deployment regimes, and manufacturing robotics use cases. The primary use cases that will be considered in this project include perception, manipulation, and robot performance monitoring/assessment. A wide range of machine learning algorithms will be considered, including deep perception models, reinforcement learning models, and other data-driven optimization approaches. Various robot-related data collection and training paradigms will also be considered, including sim-to-real transfer, learning from demonstration, and human/automated data labeling for supervised learning. Evaluating the performance of several AI-enabled robotic systems that correspond to the above paradigms, models, and use cases will serve as the basis for creating and validating useful AI productivity metrics.

Research Plan
Research objectives for this project will fall under three categories.

First, is research that helps develop and characterize robotic systems that involve key AI algorithms, training paradigms, and use cases. This includes replicating, adapting, or expanding existing research in the AI/robotics fields to address challenges that are specific to manufacturing robotics. Some examples of this include using the Segment Anything Model for bin picking segmentation or developing a novel PointNet-based model for grasp pose prediction. This research will help establish a representative set of AI-enabled robotic systems to consider for metrics development. It will also help NIST researchers better understand the practical limitations of AI in the manufacturing robotics domain. An added benefit of this research will be the creation of public-facing datasets, models, and training procedures that can immediately benefit other AI researchers or practitioners in the manufacturing robotics field.

The second objective is AI metrics development. This involves deploying AI systems in physical, production-like contexts and measuring performance using methods described above. This allows us to establish a connection between AI-related measurements and tangible physical/production outcomes. By repeating this process for many instances of AI systems, these metrics can be refined to allow for meaningful cross-system comparisons.

The third objective is the creation of guidelines, standards, and other industry-facing resources based on the AI metrics developed above. These resources will provide integrators and manufacturers who seek to implement AI-enabled robotic systems a means to make informed and calculated design decisions.

Major Accomplishments

• Developed an apparatus and a methodology for the collection of 2D and 3D data of manufacturing objects
• Disseminated a dataset of manufacturing objects and assemblies
• Established a mixed physical/simulated testbed for evaluating the performance of robot learning algorithms using two collaborative robot

Selected Publications

Merging Outcomes of SAM Applied to RGB and Depth Images in Bin Picking Applications (2024), Franaszek, Rachakonda, Piliptchak, Saidi

Qiao, H. (2023), Advanced 6D Sensor Development to Support Utilization of Cobot in High-accuracy Inspection, IEEE Proceedings of 2023 International Conference on Automation Science and Engineering, Auckland, NZ,  https://doi.org/10.1109/CASE56687.2023.10260671, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=936489 

Harrison, W., Patlolla, P., Kootbally, Z., Gupta, S.K., Using B-Splines to Measure Object Representation with Interpolative Quality in Auto-Encoders (2023) 

Kimble, K., Albrecht, J., Zimmerman, M. and Falco, J. (2022), Performance Measures to Benchmark the Grasping, Manipulation and Assembly of Deformable Objects Typical to Manufacturing Applications, Frontiers in Robotics and AI,  https://doi.org/10.3389/frobt.2022.999348, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=935275

Franaszek, M. (2022), Gauging the difficulty of image segmentation, Technical Note (NIST TN), National Institute of Standards and Technology, Gaithersburg, MD, https://doi.org/10.6028/NIST.TN.2207-upd1, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934815

Qiao, H. and Weiss, B. (2019), Industrial Robot Accuracy Degradation Monitoring and Quick Health Assessment, ASME Journal of Manufacturing Science and Engineering, https://doi.org/10.1115/1.4043649