Huang, S.
, Chen, S.
and Mott, J.
(2026),
RUFEERS (Recognizing Ultra Fine-grained Entities, Events, and Relations) Annotation Guidelines, NIST Trustworthy and Responsible AI, National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.AI.100-8, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=961716 (Accessed May 19, 2026)
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
RUFEERS (Recognizing Ultra Fine-grained Entities, Events, and Relations) Annotation Guidelines
Published
Author(s)
, Song Chen, Justin Mott
Abstract
A key area in natural language processing and by extension, in artificial intelligence, is a system's ability to extract entities (things in the world), events (who does what and when), and relations (who is related to whom in what way) and to transform unstructured data into structured data enabling automated analysis, knowledge graph construction, and deeper understanding for tasks like trend analysis, threat detection, personalized recommendations, and building medical databases by identifying who, what, where, and when in text, and the connections between them. To measure a system's performance, evaluation data must be created by humans. This document provides overall annotation guidelines for human annotators as part of the RUFEERS evaluation.Citation
NIST Trustworthy and Responsible AI - 100-8
Report Number
100-8
Pub Type
NIST Pubs
Download Paper
Keywords
data annotation, entity, entity type, event, event type, ontology, relation, relation type
Citation
Issues
If you have any questions about this publication or are having problems accessing it, please contact [email protected].
Created May 18, 2026