Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?

Published

January 26, 2022

Author(s)

Ellen M. Voorhees, Ian Soboroff, Jimmy Lin

Abstract

Neural retrieval models are generally regarded as fundamentally different from the retrieval techniques used in the late 1990's when the TREC ad hoc test collections were constructed. They thus provide the opportunity to empirically test the claim that pooling-built test collections can reliably evaluate retrieval systems that did not contribute to the construction of the collection (in other words, that such collections can be reusable). To test the reusability claim, we asked TREC assessors to judge new pools created from new search results for the TREC-8 ad hoc collection. These new search results consisted of five new runs (one each from three transformer-based models and two baseline runs that use BM25) plus the set of TREC-8 submissions that did not previously contribute to pools. The new runs did retrieve previously unseen documents, but the vast majority of those documents were not relevant. The ranking of all runs by mean evaluation score when evaluated using the official TREC-8 relevance judgment set and the newly expanded relevance set are almost identical, with Kendall's correlations greater than 0.99. Correlations for individual topics are also high. The TREC-8 ad hoc collection was originally constructed using deep pools over a diverse set of runs, including several effective manual runs. Its judgment budget, and hence construction cost, was relatively large. However, it does appear that the expense was well-spent: even with the advent of neural techniques, the collection has stood the test of time and remains a reliable evaluation instrument as retrieval techniques have advanced.

Citation

arXiv

Pub Weblink

https://arxiv.org/

Pub Type

Websites

Download Paper

https://doi.org/10.48550/arXiv.2201.11086

Local Download

Keywords

evaluation, reusability, search, TREC

Citation

Voorhees, E. , Soboroff, I. and Lin, J. (2022), Can Old TREC Collections Reliably Evaluate Modern Neural Retrieval Models?, arXiv, [online], https://doi.org/10.48550/arXiv.2201.11086, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=934130, https://arxiv.org/ (Accessed December 13, 2024)