Are Industrial AI Tools Worth it? NIST Researchers Offer An Evaluation Procedure

Industry is increasingly using artificial intelligence to aid analysis and decision-making. It is also recognizing the need to evaluate these tools with respect to their utility and value. Are these tools doing what they are supposed to, and, moreover, are they worth it? To help answer such questions and more, NIST researchers recently published a Procedural Guide for System-Level Impact Evaluation of Industrial Artificial Intelligence-Driven Technologies: Application to Risk-Based Investment Analysis for Condition Monitoring Systems in Manufacturing in the Journal of Manufacturing Science and Engineering.

The procedure focuses on evaluating condition monitoring systems, used to identify problems – faults, defects – in a system/process, and thus reduce their risks. This case study is intended to serve as a basis for evaluating other industrial artificial intelligence tools.  

The procedure begins with assessing the viability of a condition monitoring. It calls for determining if a system/process will benefit from monitoring – not all systems/processes do, notes the paper. The procedure involves determining if condition monitoring will detect problems associated with risks, assessing whether risks can be mitigated, and quantifying risks in terms of frequency and severity.  

The results help to evaluate if condition monitoring is worth it, and the paper proposes steps for this evaluation:

• Determine a system’s/process's baseline risk without condition monitoring
• Determine a condition monitoring system's installation and operating costs
• Assess the risks of operating condition monitoring system
• Estimate the value of condition monitoring for a given system/process  
• Conduct a risk-based investment analysis with metrics used by businesses

The paper also shows how these procedures could be used to evaluate condition monitoring systems in paper mill cutting and multistage laser-engraving operations.