Skip to main content

NOTICE: Due to a lapse in annual appropriations, most of this website is not being updated. Learn more.

Form submissions will still be accepted but will not receive responses at this time. Sections of this site for programs using non-appropriated funds (such as NVLAP) or those that are excepted from the shutdown (such as CHIPS and NVD) will continue to be updated.

U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

The Evolution of Cranfield

Published

Author(s)

Ellen Voorhees

Abstract

This chapter examines how the test collection paradigm, the dominant evaluation methodology in information retrieval, has been adapted to meet the changing requirements for information retrieval research in the era of community evaluation conferences such as TREC. Despite fierce criticism from the start and periodic pronouncements of its impending demise, the paradigm has enabled research that has greatly improved retrieval performance in practice. This success has largely resulted because of the paradigm's limitations rather than despite them. The document ranking task is a carefully calibrated level of abstraction that has sufficient fidelity to real user tasks to be informative, but is sufficiently abstract to be broadly applicable, feasible to implement, and comparatively inexpensive.
Citation
CLEF@20 - Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF
Publisher Info
Springer-Verlag, Berlin,

Keywords

evaluation, information retrieval, search engines, TREC

Citation

Voorhees, E. (2019), The Evolution of Cranfield, CLEF@20 - Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF, Springer-Verlag, Berlin, , [online], https://doi.org/10.1007/978-3-030-22948-1_2, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=926510 (Accessed October 13, 2025)

Issues

If you have any questions about this publication or are having problems accessing it, please contact [email protected].

Created August 14, 2019, Updated September 29, 2025
Was this page helpful?