This chapter examines how the test collection paradigm, the dominant evaluation methodology in information retrieval, has been adapted to meet the changing requirements for information retrieval research in the era of community evaluation conferences such as TREC. Despite fierce criticism from the start and periodic pronouncements of its impending demise, the paradigm has enabled research that has greatly improved retrieval performance in practice. This success has largely resulted because of the paradigm's limitations rather than despite them. The document ranking task is a carefully calibrated level of abstraction that has sufficient fidelity to real user tasks to be informative, but is sufficiently abstract to be broadly applicable, feasible to implement, and comparatively inexpensive.
CLEF@20 - Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF
The Evolution of Cranfield, CLEF@20 - Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF, Springer-Verlag, Berlin, , [online], https://doi.org/10.1007/978-3-030-22948-1_2
(Accessed November 30, 2023)