Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

LoReHLT Evaluations

Low Resource Languages for Emergent Incidents (LORELEI) is a DARPA-sponsored program. The goal of the program is to dramatically advance the state of computational linguistics and human language technology to enable rapid, low-cost development of capabilities for low-resource languages.

This web page serves as a portal for all information pertaining to the Low Resource (LoRe) HLT open evaluations of component technologies relevant to LORELEI. For general questions and comments, please e-mail lorehlt_poc [at] nist.gov (lorehlt_poc[at]nist[dot]gov).  For more information about a specific evaluation cycle, including evaluation documentation, see the year-specific sections below.  For general announcements and discussing regarding the LoReHLT evaluation series, you can subscribe to the LoReHLT mailing list hosted by NIST, lorehlt [at] nist.gov (lorehlt[at]nist[dot]gov), by sending e-mail to lorehlt_list-requeset [at] nist.gov (lorehlt_list-request[at]nist[dot]gov) with "subscribe" in the subject line.

LoReHLT19

Tasks

  • Machine translation (MT)
  • Entity Discovery and Linking (EDL)
  • Situation Frame (SF)
  • Sentiment, Emotion, & Cognitive State (SEC) (Pilot)

Highlights

  • Surprise language evaluation (two surprise languages)
  • New SEC pilot evaluation
  • Two evaluation checkpoints to gauge performance based on time and training resources given

Documentation

Tools

LoReHLT18

Tasks

  • Machine translation (MT)
  • Entity Discovery and Linking (EDL)
  • Situation Frame (SF)

Highlights

  • Surprise language evaluation (two surprise languages)
  • Two training conditions: constrained (required) and unconstrained
  • Two evaluation checkpoints to gauge performance based on time and training resources given

Registration

Documentation

Tools

Publications and System Descriptions

LoReHLT17

Tasks

  • Machine translation (MT)
  • Entity Discovery and Linking (EDL)
  • Situation Frame (SF)

Highlights

  • Surprise language evaluation (two surprise languages)
  • Two training conditions: constrained (required) and unconstrained
  • Three evaluation checkpoints to gauge performance based on time and training resources given

Documentation

Tools

Publications and System Descriptions

LoReHLT16

Tasks

  • Machine translation (MT)
  • Situation Frame (SF)
  • Named entity recognition (NER)

Highlights

  • Surprise language evaluation
  • Two training conditions: constrained (required) and unconstrained
  • Three evaluation checkpoints to gauge performance based on time and training resources given

Documentation

Tools

Publications and System Descriptions

Disclaimer

Any mention of commercial products within NIST web pages is for information only; it does not imply recommendation or endorsement by NIST.

Created November 19, 2015, Updated June 20, 2019