Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Measurement Science for Manufacturing Robotics

Summary

To develop and deploy measurement science, standards, and test methods that advance manufacturing robotic system performance, collaboration, agility, autonomy, safety, and ease of implementation to enhance U.S. innovation and industrial competitiveness.

Program videos

NIST Robotics Wares: What You Can Use or Get Involved With Now

NIST NRC PostDoc Opportunity “Measurement Science for Manufacturing Robotic Systems”

Provide your Robotics Measurement Science and Standards Needs

Description

Measurement Science for Manufacturing Robotics

Due to their tireless flexibility and reusability, robotic systems are an essential tool in strengthening U.S. manufacturing competitiveness by enabling dramatically greater responsiveness and innovation.  To attain these gains, robotic systems need to be highly-capable, perceptive, dexterous, and mobile systems that can operate safely in collaboration with humans, are easily tasked, have the ability to learn, and can be quickly integrated into the rest of the enterprise.   The program will provide the measurement science needed to enable all manufacturers, including small and medium ones, to characterize and understand the performance of robotics systems within their enterprises. Measurement science establishes a common language for expressing performance requirements and provides means of verifying that systems meet those requirements. Concrete performance targets also direct innovations towards addressing existing capability gaps in robotic systems.  NIST will deliver performance metrics, information models, data sets, test methods, and protocols to assess and assure the key attributes of robotic systems necessary to enable flexible and dynamic production.  

Objective
To develop and deploy measurement science, standards, and test methods that advance manufacturing robotic system performance, collaboration, agility, autonomy, safety, and ease of implementation to enhance U.S. innovation and industrial competitiveness.

What is the Problem?

Agile and collaborative robotic systems are a disruptive technology essential to achieving a new vision of manufacturing because of their inherent flexibility coupled with tirelessness, high-precision, and repeatability.”  Shorter product life cycles, coupled with just-in-time manufacturing, make robotic flexibility and responsiveness highly beneficial. [BCG] Yet, it is estimated that only a fraction of potential users in the manufacturing domain have adopted robotic systems.  This is because there is a lack of measurement science infrastructure to assure manufacturers that robotic components and systems can be readily integrated into their operations and will perform as needed under dynamic unstructured shop floor conditions.  [CCC]

“Improved productivity in the increasingly competitive international environment” is one of 3 key factors driving the adoption of robots cited in the Computing Community Consortium (CCC) Roadmap for U.S. Robotics. [CCC] According to the International Federation of Robotics, “the ongoing trend to automate production in order to strengthen American industries on the global market and to keep manufacturing at home, and in some cases, to bring back manufacturing that had previously been sent overseas” is driving growth in US manufacturers use of robots. [IFR] Despite the increased adoption of robotics by U. S. manufacturers, many limitations and challenges still remain.  The CCC Robotics Roadmap notes that “robotics is a key transformative technology that can revolutionize manufacturing… the promise of flexible automation and automation for mass customization has not been realized except for special cases…. Robots [need] to be smarter, more flexible, and able to operate safely in less structured environments shared with human workers.”   A workshop on Opportunities in Robotics, Automation, and Computer Science [NSF] summarized some of the barriers to greater adoption of advanced robotics by industry.  The report noted the lengthy and expensive process of installing robots as well as designing and implementing assembly lines due to lack of equipment and control models and tools.   Another key challenge is the inability to transfer successful components and solutions to other manufacturing applications or across equipment due to lack of component modularity and integration and interoperability standards.

Uninformed or under-informed performance expectations are another negative effect of the lack of measurement science.  There are serious consequences when an enterprise overestimates the capabilities of robots to handle tasks or deal with variations in the parts and environment. Many manufacturers – even cutting-edge enterprises – face challenges in automating their plants due to mismatches between expectations and robot performance. [Tesla]  
Progress towards having robots fulfill their potential within manufacturing facilities has been hindered by the lack of metrics, benchmarks, test methods, reference architectures, and standards.

Research institutions achieve advancements in sensor fusion, situational awareness, autonomous planning, grasping, navigation, and other capabilities essential to attaining a future vision of robotic systems for smart manufacturing. Several of these research advancements are powered by data-intensive machine learning and other artificial intelligence algorithms, which have been flourishing and showing promise in recent years but are difficult to evaluate and characterize.  Few of these advancements have yet made their way into commercially-available industrial robots.  The CCC Robotics Roadmap highlights the need for a measurement science infrastructure to help transition research into products and reduce the risk of adopting new robotic technologies.  This lack of measurement science infrastructure results in barriers and inefficiencies in (a) expressing end user performance requirements for new robotic capabilities; (b) assessing progress in robotic capabilities towards meeting industry needs; (c) validating new technologies for deployment by manufacturers, and (d) enabling interoperability and ease of integration of robotic systems and components.  

Why is it Hard to Solve?

The problem is difficult because next generation robots are complex multi-disciplinary systems, integrating perception, manipulation, grasping, mobility, safety, and autonomous planning subsystems that are themselves complex. Adding to the complexity is the rapid emergence of machine-learning based algorithms for perception and planning. For example, it is difficult to determine how a deep learning system arrives at a decision.  Machine learning algorithms rely on large quantities of training data, but what constitutes “good” data and best practices for its collection are yet to be determined.  The problem becomes exponentially more difficult when considering robots that collaborate with each other and with humans.  

Development of measurement science requires being able to model and predict with a high degree of confidence the capabilities of these new technologies when they are integrated into a robotic system operating within dynamic unstructured environments.  Developing methods of producing meaningful performance data that can be used in system design decisions is a major challenge. The range of potential tasks, workpieces, and environmental variability is practically limitless, further complicating the design of the performance test methods and associated measurement infrastructure and artifacts.  

A number of significant advancements are being developed for robotic systems but are not readily adoptable for integration into manufacturing robotic systems. The National Artificial Intelligence Research and Development Strategic Plan states that  “Robotics technologies are now showing promise in their ability to complement, augment, enhance, or emulate human physical capabilities or human intelligence. However, scientists need to make these robotic systems more capable, reliable, and easy-to-use.”  [AI]  New mechanisms, sensors, and materials are being explored for robot arms as well as advanced gripper designs.  To reduce the need for costly customized fixtures and grippers, novel designs that can handle a broader set of parts and tasks are starting to emerge from research laboratories. Currently, only ad hoc comparisons are possible when choosing robotic hands or to guide research towards more effective approaches. Human-robot collaboration is still in its infancy, both in terms of interaction and safety. Programming robots to perform new tasks is still typically a lengthy process requiring specialized skills. More intuitive methods, such as demonstrations by humans, may overcome the programming challenges, but there are minimal metrics and guidelines for their implementation.  Although new standards allow for robots to work near humans, extensive further research is needed to validate and verify what constitutes safe operation. Robots currently are not able to deal with uncertainty in their surroundings due to inadequate methods of modeling sensor and position errors, and lack of planning systems that can account for variations in the environment.  This is true for static arms, as well as mobile vehicles and mobile manipulators, and now wearable robots.  Manufacturers are beginning to adopt wearable robots – exoskeletons – to mitigate ergonomic challenges for their workers but there are myriad designs with unknown performance and safety implications.  Small manufacturers are particularly affected by these hurdles to implementing robotic systems since they have limited capital and expertise, and tend to have high-mix, low-volume operations that require repurposing equipment frequently. Initial NIST research has started to develop test methods in many of these areas, but additional work is needed to validate them, propose/shepherd them through the standardization process, and ensure industry adoption.

How is it Solved Today and By Whom?

Robots deployed on factory floors today require teams of specialists who have in-depth expertise for installing (the robots as well as the safety systems), calibrating, and programming them to perform specific manufacturing tasks.  Typically, this setup takes weeks and costs a multiple of the purchase price of the robot itself.   Despite yearly growth in sales, robots are utilized only in certain manufacturing operations and only by a subset of manufacturing enterprises and in a subset of manufacturing operations.  Although there has been growth in adoption by other industries, such as electronics, the biggest user of robots is still the automotive sector. [IFR]  Even so, robotics can’t be utilized in many of their operations: assembly applications, which account for about half of the production schedule in automotive manufacturing, accounted for only 7.3% of robot sales. This is due to robot’s lack of dexterity, sensing, and adaptability.  Small and medium-companies, which form the vast majority of manufacturers, have the lowest rates of adoption of robots.

The opportunities for using robotics to accelerate innovation and spur economic growth are recognized internationally. The US ranks seventh in the world in “robot density,” the number of robots per factory worker, lagging far below Korea and China. [PWC] Beyond increased robot deployments, investments in robotics research, including for manufacturing applications, has been very significant in Asia and Europe. Korea has been investing $100M per year for 10 years into robotics research and education as part of their 21st Century Frontier Program. [Sirkin]  The European Commission’s longstanding investments in robotics, including for manufacturing, were recently augmented by $900M, to be matched by triple that from industry in a new public-private partnership.[SPARC]  With the recent founding of the Advanced Robotics For Manufacturing (ARM) Institute, the U. S. has a significant public-private partnership that is seeking to advance robotic technologies in order to ensure American industry’s competitiveness. [ARM] 

Why NIST?

NIST’s mission is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life.  NIST’s role is to provide measurement science solutions to problems that hinder the progress of smart manufacturing. The program will develop and deploy measurement science, standards, and test methods that advance manufacturing robotic system performance, collaboration, agility, autonomy, safety, and ease of implementation to enhance U.S. innovation and industrial competitiveness.

The program supports the EL core competencies in intelligent systems and smart manufacturing.  Public sector involvement is necessary to overcome the initial barrier of developing the measurement science to support advancements in robotic capabilities, since the benefits will accrue broadly.  The recent focus by academics and some companies on applying machine learning and artificial intelligence to advance robotics provides a convergence of two NIST strategic priorities:  advanced manufacturing with data and AI.  EL is uniquely positioned to leverage its strong ties to industry stakeholders, academia, and standards organizations and its dedicated measurement science facilities, and build upon its sterling reputation for developing measurement science for robotic systems in manufacturing as well as response applications.  In particular, EL has been involved in advising the ARM institute on metrics and evaluation since its inception.  This partnership will provide this program with a conduit for learning about industry’s highest priority challenges and for disseminating our draft metrics, test methods, and other measurement science products.

What is the Technical Idea?

The fundamental idea is to provide the measurement science needed to ensure that robotic systems can be confidently applied to smart manufacturing operations.  Seven principal facets of robotic systems will be investigated through a holistic approach based on unified set of testbeds and scenarios in consultation with industry.  The capability-oriented research projects will be strengthened by a complementary venture focusing on reducing the technical barriers small and medium enterprises encounter today installing and using robots.   The technical idea is to develop measurement science that will characterize the performance of robotic systems and components to match appropriate solutions to implementations.  Underlying this technical idea is a process that uses application requirements to drive development of metrics and measurement science.  Following this process provides performance results that are expressed contextually, rather than as abstract quantities or qualities that may not be relevant to the intended implementation. 

Metrics, test methods, artifacts, tools, and datasets will provide assessment capabilities for

  • The verification of safe operation of robot arms and their tooling covered under the new collaborative safety standards.  
  • The effectiveness of new types of hand-like dexterous manipulators in handling a wider range of parts and operations.
  • Characterizing the mobility and safety performance of autonomous vehicles, mobile manipulators, and wearable robots within dynamic and unstructured environments.
  • The ability of robots to perceive their surroundings in order to execute tasks correctly and safely despite variations.
  • New interaction modalities between humans and robots that enable safe and efficient teaming.
  • Enhancing the agility of robots, meaning they are able to succeed in an environment of continuous and unpredictable change by reacting efficiently and effectively to changing factors.  
  • Accelerating progress in the effective utilization of artificial intelligence and machine learning through the use of validated and well-documented datasets and AI models
  • Streamlining the installation and integration of robots into workcells so that all U. S. manufacturers, including small enterprises, can reap the competitive advantages of robots.

Why Can We Succeed Now?

The manufacturing and robotics industries are at a cusp right now.   The growing recognition of the importance of automation to strengthening and accelerating U. S. manufacturing, along with the flourishing of new robotics capabilities and models, sets the stage for this program to succeed.   There is a new sense of urgency among end users – including small and medium enterprises – who are anxious to reap the benefits of flexible agile robots.  Complementary to this industry pull, there are a number of technological advancements in grasping, arms, sensors, safety, artificial intelligence, machine learning, and open source software that hold great promise for enabling robots to be much more capable, collaborative, agile, and easily integrated into manufacturing enterprises.  This is the optimal time for NIST to contribute the measurement science to ensure that these new technologies address industry’s needs.  

What is the Research Plan?

The research plan’s thrusts outlined above are addressed in the following seven projects which will share the Program’s testbeds and jointly work with industry to define relevant scenarios to drive the research.

1. Grasping, Manipulation, and Safety Performance of Robotic Systems

This thrust will provide performance metrics, test methods and associated measurement tools to support next-generation robot systems having human-like dexterity and force control characteristics that enable tactual-based safe human collaboration and manufacturing tasks.   Due to the shared underlying technologies that enable dexterity and collaborative robot safety, this research will address both thrusts.

2. Perception Performance of Robotic Systems  

The perception thrust will develop measurement science for sensing and perception system performance characterization to reduce the risk related to the adoption of new technologies and to advance the agility, safety, and productivity of collaborative industrial and mobile robots in smart manufacturing applications.

3. Mobility Performance of Robotic Systems

This thrust will provide the measurement science to develop standard test methods for performance of intelligent industrial mobility systems, including mobile robots, mobile manipulators, and wearable robots, to improve manufacturing flexibility and productivity.

4. Performance of Human-Robot Interaction  

This thrust will deliver a suite of test methods, protocols, and information models to facilitate effective human-robot collaboration in manufacturing and advance interactive robot technologies to enable manufacturers to leverage the safe and efficient teaming of people and robots toward meeting production goals.   

5. Agility Performance of Robotic Systems  

This thrust will deliver robot agility performance metrics, test methods, information models, data sets, and planning approaches that will enable manufacturers to more easily and rapidly reconfigure and re-task robot systems.

6. Embodied AI and Data Generation for Manufacturing Robotics  

This thrust will provide structured artificial intelligence (AI) and machine learning (ML) training datasets, and proven, trained, and applied AI/ML models to improve the performance and autonomy of manufacturing robotic applications.

7. Tools for Collaborative Robots within SME Workcells  

This thrust will deliver a suite of tools that facilitates calibration procedures for individual robots, robot-to-robot coordination, sensors, and grippers to mitigate the lack of automation and technical expertise that currently prevents small and medium manufacturers from adopting robotic systems.

References

[AI] National Science and Technology Council “The National Artificial Intelligence Research and Development Strategic Plan” October 2016.
[ARM] http://arminstitute.org/
[BCG]   Sander, A. and Wolfgang, M., “The Rise of Robotics,” Boston Consulting Group, August 2014.
[CCC] A Roadmap for U.S. Robotics: From Internet to Robotics, 2013.  http://www.us-robotics.us/
[Downs] Downs A, Harrison W, Schlenoff C. Test Methods for Robot Agility in Manufacturing. The Industrial robot. 2016;43(5):563-572. doi:10.1108/IR-01-2016-0032.
[IFR]  “Executive Summary World Robotics 2017 Industrial Robots,” International Federation of Robotics” 2017.|
[NSF] https://cra.org/ccc/wp-content/uploads/sites/2/2016/02/NSF-OSTP-CCC-RVO-Workshop-Report-Oct21-2013.pdf
[PWC]  PWC, Robot-ready: Adopting a new generation of industrial robots., June 2018.
[Sirkin]   Sirkin, H., Zinser, M., Rose, J., “The Robotics Revolution:  The Next Great Leap in Manufacturing, Boston Consulting Group, September 2015.
[SPARK]   http://www.sparc-robotics.net/about
[Tesla] Boudette, Neal. “Inside Tesla’s Audacious Push to Reinvent the Way Cars are Made” New York Times, June 30, 2018, https://www.nytimes.com/2018/06/30/business/tesla-factory-musk.html.

The Measurement Science for Manufacturing Robotics Program is always open to working with our external colleagues to further NIST’s mission in the areas listed on this web site, including via grants, cooperative agreements, contracts, and other more informal partnerships. Please feel free to reach out to craig.schlenoff [at] nist.gov (Craig Schlenoff) to discuss possible areas of collaboration.

Created December 11, 2018, Updated March 3, 2023