The goal of the project is to support the development of automatic systems that can provide content-based access to information that has not been explicitly structured for machine consumption. This support is in the form of evaluation infrastructure, since to quote Lord Kelvin, "If you can not measure it, you can not improve it."
The infrastructure is created through community participation in three evaluation conferences, the Text REtrieval Conference (TREC), the TREC Video Retrieval Evaluation (TRECVID), and the Text Analysis Conference (TAC). Within each conference, several focus areas are selected in collaboration with participants, outside sponsors, and other stakeholders. For each focus area, the conference provides guidelines defining the evaluation task and a corresponding data set. Participants perform the task and submit their results to NIST. For most tasks, the union of the submitted results is annotated for correctness, and these annotations form the basis for scoring individual submissions. Finally, participants gather at a workshop to discuss their results.
Open evaluations with carefully selected test data has proved to be a powerful technique to advance technology and measure progress in a field. In addition to improving the state-of-the-art, a common focus on an evaluation task forms or solidifies a research community, establishes the research methodology for the field, facilitates technology transfer, and amortizes the costs associated with building the necessary infrastructure.