Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Assembly Performance Metrics and Test Methods

Summary

New technologies in the areas of robotic arms and end-effectors have the potential to accelerate the use of robotics for assembly.  The force sensing and compliance capabilities used in collaborative robots to prevent injuries and enable them work safely alongside human workers in manufacturing environments lend themselves to robotic assembly tasks.  Additionally, robotic hands are emerging as a next generation end-effector technology with advanced force control and manipulation capabilities.  These new technologies as well as their simplified programming interfaces and reconfigurability promise new ways of tackling the small parts assembly field especially for low-volume, high-mix manufacturing operations.  A set of performance metrics, test methods and associated artifacts are being developed to progress the application of these technologies and can also be used to evaluate existing robotic assembly technologies.

Description

Performance benchmarks are being built as a modular set of task-based tests. These reconfigurable tests incorporate small part insertions and fastening methods such as threading, snap fits, and gear meshing using standard components including screws, nuts, washers, gears and electrical connectors. The test methods leverage factors identified by Boothroyd-Dewhurst (B-D) design for assembly (DFA) studies. These studies have already identified and tabulated various important factors based on manual human performance in an assembly task. For instance, size and symmetry of parts, tool usage, fixturing, mechanical resistance, mechanical fastening processes, visual occlusion, and physical obstruction are all influential towards time-based human performance. Designing benchmarking tasks that efficiently sample this design space will greatly aid the assessment of a robotic system and quickly identify its strengths and weaknesses.

Another important aspect of performance measurement is providing confidence in the measured results. Consequently, multiple repetitions of a task are required to generate enough data for benchmarking comparisons. Moreover, the use of various statistical tests including test for correlation, distribution, variance, and mean will help identify significant comparative differences in performance data.  Conducting these tests would also help reduce the number of false claims that may be issued regarding a robot’s level of performance. 

The development of assembly-relevant metrics and tests is part of a NIST-led effort sponsored by the IEEE RAS Technical Committee on Robotic Hands Grasping and Manipulation (RHGM) with the goal of identifying key competencies and characteristics of robotic systems using a robust set of formalized evaluations and benchmarks.  Ultimately, these benchmarks can help to match robotic hand capabilities to end-user needs as well as to help provide developers and researchers insight for improving their hardware and software designs. 

Current Assembly Test Designs

NIST prototype task boards are outlined below.  Although shown in vertical orientations, the task boards can be placed horizontally or in other orientations that are representative of specific assembly requirements.  Procedures for conducting the tests and capturing performance data can be found in the protocol template.

 

Task 1 Board

Task Board #1

Competencies: peg insertions, gear meshing, electrical connector insertions, nut threading

Description

This task board is designed to quantify a robot system’s capability for performing peg insertions, gear meshing, electrical connector insertions, and nut threading. Design factors include size, shape, location, and type of parts.

Benchmark Template

Replication

Instructions for reproducing the task board can be downloaded here.

Design files for laser cut board can be downloaded here.

STL files for 3D printing connector housings can be downloaded here.

CAD models for all components can be downloaded here.

STL models for all components can be downloaded here.

Kit tray for all components can be downloaded here.

Generalized test metrics, methods, and data analysis recommendations can be downloaded here.

Adoption

A similar task board was used during the IROS 2017 Robotic Grasping and Manipulation Competition.

 

The details of the competition can be found here.

 

Task Board #2

Competencies: alignment and insertion of collars and pulleys, handling flexible parts, meshing/threading belts, actuating tensioners, and threading bolts.

 

Task Board 2

Description

This task board is designed to quantify a robot system’s capability for performing alignment and insertion of collars and pulleys, handling flexible parts, meshing/threading belts, actuating tensioners, and threading bolts. Design factor: type of parts - based on commercial use and difficulty of task.

Benchmark Template

Replication

Instructions for reproducing the task board can be downloaded here

Design files for laser cut board can be downloaded here

CAD models for all components can be downloaded here

Kit tray for all components can be downloaded here

STL models for all components can be downloaded here

 

Task Board #3

Competencies: tracking, placement, weaving and manipulation of loose cables, handling flexible parts, and inserting ends into various connectors.

 

Task Board 3

Description

This task board is designed to quantify a robot system’s capability for performing tracking, placement, and weaving loose parts that require advanced manipulation in the form of hand-off and/or release and re-acquiring cables. Design factors include thickness, shape, and stiffness of cables as well as location and spacing of obstacles.

Benchmark Template

Replication

Instructions for reproducing the task board can be downloaded here

Design files for laser cut board can be downloaded here

CAD models for all components can be downloaded here (Wire Harness SLDPRTS.zip file attached)

STL models for all components can be downloaded here Parts to be 3D printed are labeled with "Custom".

Related Work

Impact of Robotic Hand on Pick-and-Place Performance

Comparative Peg-in-Hole Testing of Robotic Hand and Gripper

For performance benchmarking and test replication, please download the generated robot performance data and artifact CAD models:

Peg Hole Performance Data (zip file)

Peg Hole CAD (zip file)

Created May 23, 2018, Updated April 7, 2020