NIST’s AI Standards “Zero Drafts” Pilot Project to Accelerate Standardization, Broaden Input

NIST’s new AI Standards Zero Drafts project will pilot a process to broaden participation in and accelerate the creation of standards, helping standards meet the AI community’s needs and unleash AI innovation.

In this project, discussed at the NIST’s AI symposium in September 2024, NIST will collect input on topics with a science-backed body of work and use it to develop “zero drafts”—preliminary, stakeholder-driven drafts of standards that are as thorough as possible. These drafts then will be submitted into the private sector-led standardization process as proposals for voluntary consensus standards.

The Need

Voluntary consensus standards are typically produced by SDOs through rigorous, sometimes lengthy consensus-building processes among interested experts. As NIST has gathered input on standardization needs for AI, stakeholders have repeatedly emphasized two problems, which are especially challenging to address simultaneously:

• AI standards need to be developed expeditiously to address urgent needs, prevent fragmentation of governance frameworks, and keep up with AI advances—while still maintaining the rigor of the process and the quality of the resulting standards.
• AI standards demand a wide range of expertise and perspectives. Stakeholders seek to address many needs via AI standards. This makes it important for the standards to draw on multi-disciplinary perspectives, including from many kinds of organizations and stakeholders who develop, use, research, or are affected by AI systems.

NIST’s Solution: Convening the Community to Develop Zero Drafts

NIST seeks to expand participation in AI standards development and help standards developing organizations (SDOs) achieve consensus more quickly by releasing thorough, stakeholder-driven proposals for new SDO projects. To achieve this, NIST will collect inputs from stakeholders and craft them into detailed drafts meant to accompany new project proposals, following the process below:

1. NIST proposes topics and solicits stakeholder input on which topics to prioritize, how to scope them, and what content for each priority topic and scope NIST should include or build on.
2. For each topic selected by NIST based on this feedback:
   1. NIST releases a concept paper outlining a proposed direction for a standard;
   2. Based on the concept paper and broad stakeholder input, NIST proposes an initial draft standard;
   3. NIST iterates on drafts based on further rounds of input; and
   4. The resulting document is submitted to SDOs via established processes as a proposal for formal standardization.

Zero drafts will be thorough, high-quality documents reflecting stakeholder inputs, but they will ultimately be NIST proposals, and assuming they are taken up by SDOs, they will likely still change during the formal standardization process.

Initial Topics and Proposed Scopes 

The first several zero drafts will be approached as pilots. They will address one to two of topics below, which are based on stakeholder identified priorities expressed to NIST, and adjusted to minimize overlap with existing standards projects.

Documentation about system and data characteristics for transparency among AI actors

Potential Scopes:

1. Contents of model, data, and/or system cards, such as data collection methods, model details, performance measurements and validation thereof, and intended use.
2. Standardized mechanisms and practices for formatting, presenting, sharing, and/or accessing documentation.
3. Application of existing documentation practices for securing information technology supply chains to AI systems.

Methods and metrics for AI testing, evaluation, verification, and validation (TEVV)

Potential Scopes:

1. Application of well-established TEVV methods (e.g., for establishing construct validity, performing sensitivity analysis, or field testing) to TEVV for generative AI.
2. Approaches for translating heterogeneous benchmark scores into meaningful scores or rankings for a given use case.
3. Methods for preventing training data from becoming “contaminated” with canonical test outputs and detecting when contamination has occurred.

Maps of concepts and terminology regarding AI system designs, architectures, processes, and actors

Potential Scopes:

1. Clarification of the “AI stack”—the layers of technology and resources used to build AI applications, including the roles, responsibilities, and processes involved in each layer across the AI lifecycle.
2. Reference architectures or design patterns for AI systems to establish shared understanding of AI system components and their relationships. 

Technical measures for reducing risks posed by synthetic content

Potential Scopes:

1. A taxonomy of approaches and terms to refer to these approaches (e.g., digital content transparency, provenance data tracking, signed metadata, watermarking).
2. Methods and metrics for evaluating and reporting the effectiveness of such measures.