2024 NIST Generative AI (GenAI): Data Creation Specification for Text-to-Text (T2T) Generators

Published

April 1, 2024

Author(s)

Yooyoung Lee, George Awad, Asad Butt, Lukas Diduch, Kay Peterson, Seungmin Seo, Ian Soboroff, Hariharan Iyer

Abstract

Generator (G) teams will be tested on their system ability to generate content that is indistinguishable from human-generated content. For the pilot study, the evaluation will help determine strengths and weaknesses in their approaches including insights about how and when humans and/or AI can detect AI-generated content. Discriminator (D) teams will be tested on their system ability to differentiate between AI-generated content and human-generated content which has recently been a major research challenge, such as source for mis/disinformation and trustworthy information in digital content. Lessons learned from both sides of teams should benefit future research directions and approaches to understand cutting-edge technologies as well as source for recommendations and guidance for responsible and safe use of digital content. The pilot study in 2024 GenAI evaluations will focus on text modality. The pilot GenAI generator task, an objective of Text-to-Text Generators (T2T-G) is to generate automated high quality summaries given a topic of information need and set of source documents to utilize. On the other hand, the pilot Text-to-Text Discriminators (T2T-D) task is to detect if a target output summary has been generated using Generative AI system or a Human. The context of this challenge assumes complete AI-generated content (ignoring cases where humans use AI tools to co-author content such as rephrasing, grammar correction, editing, etc.).

Citation

Generative AI challenge

Pub Weblink

https://ai-challenges.nist.gov/genai

Pub Type

Websites

Download Paper

Local Download

Keywords

Generative AI, Large Language Model (LLM), evaluation, challenge, performance measure, text-to-text, text-to-image

Citation

Lee, Y. , Awad, G. , Butt, A. , Diduch, L. , Peterson, K. , Seo, S. , Soboroff, I. and Iyer, H. (2024), 2024 NIST Generative AI (GenAI): Data Creation Specification for Text-to-Text (T2T) Generators, Generative AI challenge, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=957327, https://ai-challenges.nist.gov/genai (Accessed October 8, 2025)