Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

AI Metrology Colloquia Series

As a follow-on to the National Academies of Science, Engineering, and Medicine workshop on Assessing and Improving AI Trustworthiness (link) and the National Institute of Standards and Technology (NIST) workshop on AI Measurement and Evaluation (link), NIST has begun hosting a bi-weekly AI metrology colloquia series, where leading researchers share current and recent work in AI measurement and evaluation.

This series provide a dedicated venue for the presentation and discussion of AI metrology research and to spur collaboration among AI metrology researchers in order to help advance the state-of-the-art in AI measurement and evaluation. The series is open to the public and the presentation formats are flexible, though generally consist of 50-minute talks with 10 minutes of questions and discussion.

Please contact aime [at] nist.gov with any questions, or to join the AIME mailing list.

Summer/Fall 2021 Schedule

Date

Presenter

Title

July 29

Ellen Voorhees, NIST

Operationalizing Trustworthy AI

August 12

Michael Majurski, NIST

Trojan Detection Evaluation: Finding Hidden Behavior in AI Models

August 26

Thurston Sexton, NIST

Understanding & Evaluating Informed NLP Systems: The Road to Technical Language Processing

September 9

José Hernández-Orallo, Universitat Politècnica de València / Leverhulme Centre for the Future of Intelligence (Cambridge)

Measuring Capabilities and Generality in Artificial Intelligence

September 23

Jonathon Phillips, NIST

Face Recognition: from Evaluations to Experiment

October 7

---

 

October 21

Finale Doshi-Velez, Harvard

The Promises, Pitfalls, and Validation of Explainable AI

November 4

Michael Sharp, NIST

Risk Management in Industrial Artificial Intelligence

November 18

David Kanter, ML Commons

Introduction to MLCommons and MLPerf

December 2

Andreas Holzinger, Medical University Graz / Graz University of Technology / University of Alberta

Assessing and Improving AI Trustworthiness with the Systems Causability Scale

December 16

Rich Kuhn, NIST

How Can We Provide Assured Autonomy?

Winter/Spring 2022 Schedule

Date

Presenter

Title

January 27

Brian Stanton, NIST

Trust and Artificial Intelligence 

February 10

Timnit Gebru, Distributed AI Research Institute (DAIR)

DAIR & AI and Its Consequences

February 24

Peter Hase, UNC

Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?

March 10

Dan Weld, Allen Institute for AI (AI2)

Optimizing Human-AI Teams

March 24

Haiying Guan, NIST

Open Media Forensic Challenge (OpenMFC) Evaluation Program

April 7 Rayid Ghani, Professor in Machine Learning and Public Policy at Carnegie Mellon University Practical Lessons and Challenges in Building Fair and Equitable AI/ML Systems
April 21 Dr. Yuekai Sun, Statistics Department at the University of Michigan Statistical Perspectives on Federated Learning
May 5 Dr. Judy Hoffman, Georgia Tech Measuring and Mitigating Bias in Vision Systems
May 19 Jonathan Fiscus, NIST  The Activities in Extended Video Evaluations : A Case Study in AI Metrology
June 2 Reva Schwartz, Apostol Vassilev, Kristen Greene & Lori A. Perine / National Institute of Standards and Technology Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST Special Publication 1270)
June 16 Theodore Jensen / National Institute of Standards and Technology

User Trust Appropriateness in Human-AI Interaction

July 14

Andrew Trask, OpenMined / University of Oxford / Centre for the Governance of AI

Privacy-preserving AI 
July 28 Nicholas Carlini / Google Brain Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples
August 11 Chunyuan Li / Microsoft Research

A Vision-Language Approach to Computer Vision in the Wild: Modeling and Benchmarking

August 25 Aylin Caliskan and Robert Wolfe / University of Washington Quantifying Biases and Societal Defaults in Word Embeddings and Language-Vision AI
Sep 8 Been Kim / Google Brain Bridging the representation gap between humans and machines: first steps
Sep 22 Douwe Kiela / Head of Research at Hugging Face

Rethinking benchmarking in AI: Evaluation-as-a-Service and Dynamic Adversarial Data Collection

Oct 6 Thomas Dietterich / Oregon State University Methodological Issues in Anomaly Detection Research
Oct 20 Soheil Feizi TBD

Portions of the events may be recorded and audience Q&A or comments may be captured. The recorded event may be edited and rebroadcast or otherwise made publicly available by NIST.  By registering for -- or attending -- this event, you acknowledge and consent to being recorded.

 

Created March 15, 2022, Updated September 22, 2022