Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

2024 NIST Generative AI (GenAI): Data Creation Specification for Text-to-Text (T2T) Generators

Published

Author(s)

Yooyoung Lee, George Awad, Asad Butt, Lukas Diduch, Kay Peterson, Seungmin Seo, Ian Soboroff, Hariharan Iyer

Abstract

Generator (G) teams will be tested on their system ability to generate content that is indistinguishable from human-generated content. For the pilot study, the evaluation will help determine strengths and weaknesses in their approaches including insights about how and when humans and/or AI can detect AI-generated content. Discriminator (D) teams will be tested on their system ability to differentiate between AI-generated content and human-generated content which has recently been a major research challenge, such as source for mis/disinformation and trustworthy information in digital content. Lessons learned from both sides of teams should benefit future research directions and approaches to understand cutting-edge technologies as well as source for recommendations and guidance for responsible and safe use of digital content. The pilot study in 2024 GenAI evaluations will focus on text modality. The pilot GenAI generator task, an objective of Text-to-Text Generators (T2T-G) is to generate automated high quality summaries given a topic of information need and set of source documents to utilize. On the other hand, the pilot Text-to-Text Discriminators (T2T-D) task is to detect if a target output summary has been generated using Generative AI system or a Human. The context of this challenge assumes complete AI-generated content (ignoring cases where humans use AI tools to co-author content such as rephrasing, grammar correction, editing, etc.).
Citation
Generative AI challenge

Keywords

Generative AI, Large Language Model (LLM), evaluation, challenge, performance measure, text-to-text, text-to-image

Citation

Lee, Y. , Awad, G. , Butt, A. , Diduch, L. , Peterson, K. , Seo, S. , Soboroff, I. and Iyer, H. (2024), 2024 NIST Generative AI (GenAI): Data Creation Specification for Text-to-Text (T2T) Generators, Generative AI challenge, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=957327, https://ai-challenges.nist.gov/genai (Accessed February 18, 2025)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created April 1, 2024, Updated January 28, 2025