NIST Information Technology Laboratory AI Webinar Series: Building Measurement Probes into Agentic AI Ecosystems

AI agents represent the next evolution of language models, capable of planning multi-step tasks, utilizing tools, searching databases, and making decisions autonomously. As organizations integrate these agents into higher-stakes ecosystems, there is an urgent need for measurement tools that can trace the decisions agentic models make and verify their appropriateness. Traceability depends upon the ability to reconstruct the chain of reasoning, usage of tools, and other evidence behind an agent's decision.

The NIST Information Technology Laboratory (ITL) AI Program will host a technical webinar on early research focused on developing automated measurement tools, called probes, to build traceability into agentic AI ecosystems. The approach adapts established techniques such as judges/verifiers, grounds them in a knowledge-base, and empowers them to evaluate agentic AI output.

The webinar will discuss existing technical gaps in the measurement infrastructure of agentic AI systems. The team will outline initial research into a promising approach for applying concepts such as adversarial verifiers to better evaluate Agentic AI outputs.

During and following the webinar, participants will be encouraged to identify practical measurement challenges they face with agentic AI. The ITL team is seeking suggestions for potential use cases – including but not limited to those which are domain-specific – that would benefit from dynamic verification and help refine and expand the development of automated measurement of agentic AI systems.