Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Are Industrial AI Tools Worth it? NIST Researchers Offer An Evaluation Procedure

How do you know your getting your money's worth from industrial AI

Industry is increasingly using artificial intelligence to aid analysis and decision-making. It is also recognizing the need to evaluate these tools with respect to their utility and value. Are these tools doing what they are supposed to, and, moreover, are they worth it? To help answer such questions and more, NIST researchers recently published a Procedural Guide for System-Level Impact Evaluation of Industrial Artificial Intelligence-Driven Technologies: Application to Risk-Based Investment Analysis for Condition Monitoring Systems in Manufacturing in the Journal of Manufacturing Science and Engineering.

The procedure focuses on evaluating condition monitoring systems, used to identify problems – faults, defects – in a system/process, and thus reduce their risks. This case study is intended to serve as a basis for evaluating other industrial artificial intelligence tools.  

The procedure begins with assessing the viability of a condition monitoring. It calls for determining if a system/process will benefit from monitoring – not all systems/processes do, notes the paper. The procedure involves determining if condition monitoring will detect problems associated with risks, assessing whether risks can be mitigated, and quantifying risks in terms of frequency and severity.  

The results help to evaluate if condition monitoring is worth it, and the paper proposes steps for this evaluation:

  • Determine a system’s/process's baseline risk without condition monitoring
  • Determine a condition monitoring system's installation and operating costs
  • Assess the risks of operating condition monitoring system
  • Estimate the value of condition monitoring for a given system/process  
  • Conduct a risk-based investment analysis with metrics used by businesses

The paper also shows how these procedures could be used to evaluate condition monitoring systems in paper mill cutting and multistage laser-engraving operations. 

Released February 1, 2022, Updated June 26, 2022