Physical AI and Data Generation for Robotics

Summary

Artificial intelligence (AI) in the form of advanced machine learning models has been widely adopted in many technology applications. These machine learning models and techniques show great potential for improving the performance of robotic systems but pose unique challenges to both the machine learning and robotics aspects of the system. This is especially true in the manufacturing domain, where the cost, safety, and productive output of the robot system are of high importance. Currently, a large gap exists between embodied AI seen in academic research and what is feasible to implement in the real-world by manufacturers and robotic systems integrators. The goal of this project is to help bridge this gap by implementing and characterizing various embodied AI approaches that are relevant and practical to manufacturing robot system developers. As part of this process, the project will develop test methods assessing the performance of AI-enabled robot systems and communicate these results to industry via guidelines, standards, reports, publications, and/or datasets. The manufacturing robotics community stands to benefit from this work by gaining new methods for assessing the feasibility of AI-based solutions to industry challenges, and the academic community stands to benefit through access to industry collaborators to facilitate the practical adoption of their research. NIST is uniquely qualified to address this because of our experience with robot system measurement and test method development, as well as our wide range of contacts across industry and academia for facilitating technology transfer and collaborations.

Description

Credit: Helen Qiao and Rutvik Kevadiya (NIST)

Objective
Develop metrics, test methods, standards, software, prototypes, and datasets to promote the adoption of AI-enhanced robotics.

Technical Idea
To address this problem, manufacturers and integrators need a streamlined way of assessing the productive impact of AI systems. To address the first need, NIST plans to develop AI-specific productivity metrics and test methods. These AI metrics will be built on the typical AI-development pipeline involving data collection, data pre-processing, training, and deployment. While the particulars of each of these steps may differ depending on the algorithm and application, the costs incurred at each step can be concretely defined and characterized. Furthermore, the productive benefits of the AI system can also be measured with the help of general machine learning metrics such as model accuracy, precision/recall, and mean average precision (mAP). The technical challenge of developing these metrics lies in understanding the relationship between AI algorithm, robot system, and task as well as their combined effects on cost/performance. Because there are countless ways to combine these three elements, it is crucial for these NIST-developed metrics to consider a representative set of AI-enabled robot systems to make sure these metrics adequately cover the range of possible data collection modalities, machine learning algorithms, training/deployment regimes, and manufacturing robotics use cases. To build this representative set of robot systems NIST will rely on in-house expertise, cooperative agreements, and other collaborations with manufacturing industry partners. The publications, datasets, software, and prototypes developed as part of this effort will be made publicly available and will serve as intermediate products that support the metrics-development process. As this process matures, metrics and test methods developed by NIST will be formalized and incorporated into AI robotics standards. Along with metrics-development, NIST will concurrently foster engagement from our collaborators and stakeholders in order to establish a wider consensus for these metrics, thus increasing the impact of any final standards that emerge.

Research Plan
The research plan for this project consists of four general directions in support of the products described above: 1) the development of AI metrics and evaluation methodologies, 2) the transfer of AI technology to the manufacturing industry, 3) expanding the applicability of AI to manufacturing, and 4) standards development activities.

1. Research that involves evaluating AI models in a manufacturing robotics context will directly be used to create formal metrics encompassing a broad range of AI models and applications. Models that are to be evaluated are selected from a variety of technical robotics areas, including models for perception, manipulation, and performance monitoring.

2. The “technology transfer” direction refers to any research that involves creating or supporting the development of AI models that are suitable for industrial deployment. It is critical that NIST metrics are designed for viable real-world AI models to maximize their relevance to industry. Because AI in manufacturing robotics is still an emerging area, any software, datasets, and publications developed as part of these efforts will be of high value as references and example implementations that industry can leverage in their own research and development.

3. As new AI technologies emerge, it is important for NIST to understand their full range of application to manufacturing robotics. While past work has focused on simpler pick-and-place grasping applications, we are now able to research and test use cases in assembly, drilling, dexterous manipulation, and more. These new use cases will be incorporated into the development of metrics and test methods to give them broader coverage.

4. The final direction is the creation of guidelines, standards, and other industry-facing resources based on NIST-developed AI metrics and standards. These resources will provide integrators and manufacturers who seek to implement AI-enabled robot systems a means to make informed and calculated design decisions.

---

Major Accomplishments

• Developed an apparatus and a methodology for the collection of 2D and 3D data of manufacturing objects
• Disseminated a dataset of manufacturing objects and assemblies
• Established a mixed physical/simulated testbed for evaluating the performance of robot learning algorithms using two collaborative robot

Selected Publications

Merging Outcomes of SAM Applied to RGB and Depth Images in Bin Picking Applications (2024), Franaszek, Rachakonda, Piliptchak, Saidi

Qiao, H. (2023), Advanced 6D Sensor Development to Support Utilization of Cobot in High-accuracy Inspection, IEEE Proceedings of 2023 International Conference on Automation Science and Engineering, Auckland, NZ,  https://doi.org/10.1109/CASE56687.2023.10260671, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=936489 

Harrison, W., Patlolla, P., Kootbally, Z., Gupta, S.K., Using B-Splines to Measure Object Representation with Interpolative Quality in Auto-Encoders (2023) 

Kimble, K., Albrecht, J., Zimmerman, M. and Falco, J. (2022), Performance Measures to Benchmark the Grasping, Manipulation and Assembly of Deformable Objects Typical to Manufacturing Applications, Frontiers in Robotics and AI,  https://doi.org/10.3389/frobt.2022.999348, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=935275

Franaszek, M. (2022), Gauging the difficulty of image segmentation, Technical Note (NIST TN), National Institute of Standards and Technology, Gaithersburg, MD, https://doi.org/10.6028/NIST.TN.2207-upd1, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934815

Qiao, H. and Weiss, B. (2019), Industrial Robot Accuracy Degradation Monitoring and Quick Health Assessment, ASME Journal of Manufacturing Science and Engineering, https://doi.org/10.1115/1.4043649